Version 323

This commit is contained in:
Hydrus Network Developer 2018-09-19 16:54:51 -05:00
parent 817b4f2fb6
commit 5ed6db8503
40 changed files with 1571 additions and 355 deletions

View File

@ -7,7 +7,13 @@
<body>
<div class="content">
<h3>all downloaders are user-creatable and -shareable</h3>
<p>Under construction! I haven't got easy import going yet!</p>
<p>Since the big downloader overhaul, all downloaders can be created, edited, and shared by any user. Creating one from scratch is not simple, and it takes a little technical knowledge, but importing what someone else has created is easy.</p>
<p>Hydrus objects like downloaders can sometimes be shared as data encoded into png files, like this:</p>
<p><img src="safebooru_downloader.png" /></p>
<p>This contains all the information needed for a client to add a 'safebooru tag search' entry to the list you select from when you start a new download or subscription.</p>
<p>You can get these pngs from anyone who has experience in the downloader system. I expect an archive to appear <a href="https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System">here</a>.</p>
<p>To 'add' these pngs to your client, hit <i>network->downloaders->import downloaders</i>. A little image-panel will appear onto which you can drag-and-drop these png files. The client will then decode and go through the png, looking for interesting new objects and automatically import and link them up without you having to do any more. Your only further input on your end is a 'does this look correct?' check right before the actual import, just to make sure there isn't some mistake or other glaring problem.</p>
<p>Objects imported this way will take precedence over existing functionality, so if your safebooru downloader were to stop working due to some site change and someone gave you a new fixed png, drag-and-dropping the png <i>should</i> import and move to using the the new updated downloader components.</i>
</div>
</body>
</html>

View File

@ -8,6 +8,32 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 323</h3></li>
<ul>
<li>wrote first version of the new downloader easy-import drop-panel. you drop downloader-encoded pngs on it, and it maybe asks you a question and jumbles its way through auto-importing all the required data to the client</li>
<li>extended this file import to do some cleverer 'example url merging' when parsers are otherwise dupes, rather than spamming similar dupes on import</li>
<li>wrote first version of the new downloader export panel. it takes gugs, url classes and parsers, and predicts sensible sub-objects to include to make functional downloaders, and bundles it into one png</li>
<li>fleshed out help for the new easy import/export system</li>
<li>the client now slows down gallery and watcher processing when the network engine is under heavy load, aiming for no more than 50 jobs in system at once. the solution is a bit hacky for now, but it should alleviate the deadlock issue when there are ~180+ simultaneous gallery/watcher network jobs pending</li>
<li>the multi-watcher panel's list of watchers now supports right-click menu to copy/open urls and pause/play files/checking</li>
<li>the multi-downloader panel's list of downloaders now supports right-click menu to copy query texts and pause/play files/searching</li>
<li>added a 'derpibooru tag search - no filter' GUG that disables the default derpi no-explicit-files rule</li>
<li>added basic gfycat support to default client--drag and drop any typical video page, and it should import ok</li>
<li>fixed the canvas/hover window tag sorting discrepancy--all tags are now sorted with the same code, and the media view sort order should be the same as your default sort order (although in this case incidence has no effect as there are no tag counts)</li>
<li>rewrote the network job control's cog menu to be a bit more dynamic, and added 'override gallery slot requirements for this job' if appropriate</li>
<li>fixed a stupid typo bug in the shutdown maintenance jobs test code that was causing pending repository work to not report right</li>
<li>fixed gallery searches that include unicode characters that end up in the path of the url (rather than the query parameters)</li>
<li>fixed an issue where highlighting a watcher would unpause its checking</li>
<li>generalised the way the new listctrl class can produce right-click menus</li>
<li>fixed some api link calculation that was over-prescribing api link display pairs (this affected the artstation file page url class by default). these pairs are now also sorted in the links dialog</li>
<li>misc png-export improvements to present better with the new easy import/export stuff</li>
<li>the summary texts in the tag filter panel now ellipsize (...), so if the tag filter is complicated, it won't try to boot a superwide edit panel!</li>
<li>the manage subscriptions panel now correctly initially sorts in a case-insensitive way (previously, it was usually sorting A-Za-z, which is different to regular aA-zZ resorting behaviour, so it always sort-flickered after the first edit)</li>
<li>the status bar has a new segment for reporting when the client is 'busy' with different jobs. for most typical usage, it'll just stay blank. let's see how it goes.</li>
<li>fixed mr. bones's wild review when the client currently has no files</li>
<li>punched up the new file report mode to specify full paths where available</li>
<li>improved some misc downloader code</li>
</ul>
<li><h3>version 322</h3></li>
<ul>
<li>wrote gugs help</li>

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

View File

@ -8,6 +8,7 @@
<div class="content">
<p><a href="downloader_sharing.html"><---- Back to sharing</a></p>
<h3>login</h3>
<p>This is not done yet!</p>
<p class="right"><a href="index.html">Back to the index ----></a></p>
</div>
</body>

View File

@ -8,7 +8,13 @@
<div class="content">
<p><a href="downloader_completion.html"><---- Back to putting downloaders together</a></p>
<h3>sharing</h3>
<p>Under construction! I haven't got this going yet!</p>
<p>If you are working with users who also understand the downloader system, you can swap your GUGs, URL Classes, and Parsers separately using the import/export buttons on the relevant dialogs, which work in pngs and clipboard text.</p>
<p>But if you want to share conveniently, and with users who are not familiar with the different downloader objects, you can package everything into a single easy-import png as per <a href="adding_new_downloaders.html">here</a>.</p>
<p>The dialog to use is <i>network->downloader definitions->export downloaders</i>:</p>
<p><img src="downloader_export_panel.png" /></p>
<p>It isn't difficult. Essentially, you want to bundle enough objects to make one or more 'working' GUGs at the end. I recommend you start by just hitting 'add gug', which--using Example URLs--will attempt to figure out everything you need by itself.</p>
<p>This all works on Example URLs and some domain guesswork, so make sure your url classes are good and the parsers have correct Example URLs as well. If they don't, they won't all link up neatly for the end user. If part of your downloader is on a different domain to the GUGs and Gallery URLs, then you'll have to add them manually. Just start with 'add gug' and see if it looks like enough.</p>
<p>Once you have the necessary and sufficient objects added, you can export to png. You'll get a similar 'does this look right?' summary as what the end-user will see, just to check you have everything in order and the domains all correct. If that is good, then make sure to give the png a sensible filename and embellish the title and description if you need to. You can then send/post that png wherever, and any regular user will be able to use your work.</p>
<p class="right"><a href="downloader_login.html">Onto the login manager ----></a></p>
</div>
</body>

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.1 KiB

View File

@ -687,13 +687,13 @@ class ClientFilesManager( object ):
def LocklessAddFileFromString( self, hash, mime, data ):
dest_path = self._GenerateExpectedFilePath( hash, mime )
if HG.file_report_mode:
HydrusData.ShowText( 'Adding file from string: ' + HydrusData.ToUnicode( ( hash, mime, HydrusData.ToByteString( len( data ) ) ) ) )
HydrusData.ShowText( 'Adding file from string: ' + HydrusData.ToUnicode( ( HydrusData.ToByteString( len( data ) ), dest_path ) ) )
dest_path = self._GenerateExpectedFilePath( hash, mime )
HydrusPaths.MakeFileWritable( dest_path )
with open( dest_path, 'wb' ) as f:
@ -704,13 +704,13 @@ class ClientFilesManager( object ):
def LocklessAddFile( self, hash, mime, source_path ):
dest_path = self._GenerateExpectedFilePath( hash, mime )
if HG.file_report_mode:
HydrusData.ShowText( 'Adding file from path: ' + HydrusData.ToUnicode( ( hash, mime, source_path ) ) )
HydrusData.ShowText( 'Adding file from path: ' + HydrusData.ToUnicode( ( source_path, dest_path ) ) )
dest_path = self._GenerateExpectedFilePath( hash, mime )
if not os.path.exists( dest_path ):
successful = HydrusPaths.MirrorFile( source_path, dest_path )
@ -732,16 +732,16 @@ class ClientFilesManager( object ):
def LocklessAddFullSizeThumbnail( self, hash, thumbnail ):
dest_path = self._GenerateExpectedFullSizeThumbnailPath( hash )
if HG.file_report_mode:
HydrusData.ShowText( 'Adding full-size thumbnail: ' + HydrusData.ToUnicode( ( hash, HydrusData.ToByteString( len( thumbnail ) ) ) ) )
HydrusData.ShowText( 'Adding full-size thumbnail: ' + HydrusData.ToUnicode( ( HydrusData.ToByteString( len( thumbnail ) ), dest_path ) ) )
path = self._GenerateExpectedFullSizeThumbnailPath( hash )
HydrusPaths.MakeFileWritable( dest_path )
HydrusPaths.MakeFileWritable( path )
with open( path, 'wb' ) as f:
with open( dest_path, 'wb' ) as f:
f.write( thumbnail )
@ -1072,6 +1072,11 @@ class ClientFilesManager( object ):
path = self._GenerateExpectedFilePath( hash, mime )
if HG.file_report_mode:
HydrusData.ShowText( 'File path request success: ' + HydrusData.ToUnicode( path ) )
if check_file_exists and not os.path.exists( path ):
raise HydrusExceptions.FileMissingException( 'No file found at path + ' + path + '!' )
@ -1103,21 +1108,26 @@ class ClientFilesManager( object ):
if HG.file_report_mode:
HydrusData.ShowText( 'Full-size thumbnail path request success: ' + HydrusData.ToUnicode( path ) )
return path
def GetResizedThumbnailPath( self, hash, mime ):
if HG.file_report_mode:
HydrusData.ShowText( 'Resized thumbnail path request: ' + HydrusData.ToUnicode( ( hash, mime ) ) )
with self._lock:
path = self._GenerateExpectedResizedThumbnailPath( hash )
if HG.file_report_mode:
HydrusData.ShowText( 'Resized thumbnail path request: ' + HydrusData.ToUnicode( path ) )
if not os.path.exists( path ):
self._GenerateResizedThumbnail( hash, mime )
@ -1129,13 +1139,13 @@ class ClientFilesManager( object ):
def LocklessHasFullSizeThumbnail( self, hash ):
path = self._GenerateExpectedFullSizeThumbnailPath( hash )
if HG.file_report_mode:
HydrusData.ShowText( 'Full-size thumbnail path test: ' + HydrusData.ToUnicode( hash ) )
HydrusData.ShowText( 'Full-size thumbnail path test: ' + HydrusData.ToUnicode( path ) )
path = self._GenerateExpectedFullSizeThumbnailPath( hash )
return os.path.exists( path )

View File

@ -527,7 +527,7 @@ class Controller( HydrusController.HydrusController ):
if service.CanDoIdleShutdownWork():
work_to_do.append( service.GetName + ' repository processing' )
work_to_do.append( service.GetName() + ' repository processing' )

View File

@ -10922,6 +10922,44 @@ class DB( HydrusDB.HydrusDB ):
if version == 322:
try:
domain_manager = self._GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER )
domain_manager.Initialise()
#
domain_manager.OverwriteDefaultGUGs( [ 'derpibooru tag search - no filter' ] )
#
domain_manager.OverwriteDefaultURLMatches( [ 'gfycat file page', 'gfycat file page api', 'gfycat file page - gif url', 'gfycat file page - detail url' ] )
#
domain_manager.OverwriteDefaultParsers( [ 'gfycat file page api parser' ] )
#
domain_manager.TryToLinkURLMatchesAndParsers()
#
self._SetJSONDump( domain_manager )
except Exception as e:
HydrusData.PrintException( e )
message = 'Trying to update some url classes and parsers failed! Please let hydrus dev know!'
self.pub_initial_message( message )
self._controller.pub( 'splash_set_title_text', 'updated db to v' + str( version + 1 ) )
self._c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )

View File

@ -79,13 +79,14 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
bandwidth_width = ClientGUICommon.ConvertTextToPixelWidth( self, 17 )
idle_width = ClientGUICommon.ConvertTextToPixelWidth( self, 6 )
hydrus_busy_width = ClientGUICommon.ConvertTextToPixelWidth( self, 11 )
system_busy_width = ClientGUICommon.ConvertTextToPixelWidth( self, 13 )
db_width = ClientGUICommon.ConvertTextToPixelWidth( self, 14 )
stb_style = wx.STB_SIZEGRIP | wx.STB_ELLIPSIZE_END | wx.FULL_REPAINT_ON_RESIZE
self._statusbar = self.CreateStatusBar( 5, stb_style )
self._statusbar.SetStatusWidths( [ -1, bandwidth_width, idle_width, system_busy_width, db_width ] )
self._statusbar = self.CreateStatusBar( 6, stb_style )
self._statusbar.SetStatusWidths( [ -1, bandwidth_width, idle_width, hydrus_busy_width, system_busy_width, db_width ] )
self._statusbar_thread_updater = ClientGUICommon.ThreadToGUIUpdater( self._statusbar, self.RefreshStatusBar )
@ -1109,6 +1110,18 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _ExportDownloader( self ):
with ClientGUITopLevelWindows.DialogNullipotent( self, 'export downloaders' ) as dlg:
panel = ClientGUIParsing.DownloaderExportPanel( dlg, self._controller.network_engine )
dlg.SetPanel( panel )
dlg.ShowModal()
def _FetchIP( self, service_key ):
with ClientGUIDialogs.DialogTextEntry( self, 'Enter the file\'s hash.' ) as dlg:
@ -1713,7 +1726,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
# this will be the easy-mode 'export ability to download from blahbooru' that'll bundle it all into a nice package with a neat png.
# need a name for this that isn't 'downloader', or maybe it should be, and I should rename downloaders below to 'gallery query generator' or whatever.
ClientGUIMenus.AppendMenuLabel( submenu, 'UNDER CONSTRUCTION: review and import/export downloaders', 'Review where you can download from and import or export that data in order to share with other users.' )
ClientGUIMenus.AppendMenuItem( self, submenu, 'import downloaders', 'Import new download capability through encoded pngs from other users.', self._ImportDownloaders )
ClientGUIMenus.AppendSeparator( submenu )
@ -1733,6 +1746,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
ClientGUIMenus.AppendSeparator( submenu )
ClientGUIMenus.AppendMenuItem( self, submenu, 'manage url class links', 'Configure how URLs present across the client.', self._ManageURLMatchLinks )
ClientGUIMenus.AppendMenuItem( self, submenu, 'export downloaders', 'Export downloader components to easy-import pngs.', self._ExportDownloader )
ClientGUIMenus.AppendSeparator( submenu )
@ -2038,6 +2052,15 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
frame.SetPanel( panel )
def _ImportDownloaders( self ):
frame = ClientGUITopLevelWindows.FrameThatTakesScrollablePanel( self, 'import downloaders' )
panel = ClientGUIScrolledPanelsReview.ReviewDownloaderImport( frame, self._controller.network_engine )
frame.SetPanel( panel )
def _ImportFiles( self, paths = None ):
if paths is None:
@ -2905,9 +2928,11 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
idle_status = ''
hydrus_busy_status = self._controller.GetThreadPoolBusyStatus()
if self._controller.SystemBusy():
busy_status = 'system busy'
busy_status = 'CPU busy'
else:
@ -2920,8 +2945,9 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
self._statusbar.SetStatusText( media_status, 0 )
self._statusbar.SetStatusText( idle_status, 2 )
self._statusbar.SetStatusText( busy_status, 3 )
self._statusbar.SetStatusText( db_status, 4 )
self._statusbar.SetStatusText( hydrus_busy_status, 3 )
self._statusbar.SetStatusText( busy_status, 4 )
self._statusbar.SetStatusText( db_status, 5 )
def _RegenerateACCache( self ):

View File

@ -2630,7 +2630,7 @@ class CanvasWithDetails( Canvas ):
tags_i_want_to_display = list( tags_i_want_to_display )
ClientTags.SortTagsList( tags_i_want_to_display, HC.options[ 'default_tag_sort' ] )
ClientTags.SortTags( HC.options[ 'default_tag_sort' ], tags_i_want_to_display )
current_y = 3

View File

@ -551,25 +551,7 @@ class NetworkJobControl( wx.Panel ):
self._gauge = ClientGUICommon.Gauge( self )
menu_items = []
invert_call = self.FlipOverrideBandwidthForCurrentJob
value_call = self.CurrentJobOverridesBandwidth
check_manager = ClientGUICommon.CheckboxManagerCalls( invert_call, value_call )
menu_items.append( ( 'check', 'override bandwidth rules for this job', 'Tell the current job to ignore existing bandwidth rules and go ahead anyway.', check_manager ) )
menu_items.append( ( 'separator', 0, 0, 0 ) )
invert_call = self.FlipAutoOverrideBandwidth
value_call = self.AutoOverrideBandwidth
check_manager = ClientGUICommon.CheckboxManagerCalls( invert_call, value_call )
menu_items.append( ( 'check', 'auto-override bandwidth rules for all jobs here after five seconds', 'Ignore existing bandwidth rules for all jobs under this control, instead waiting a flat five seconds.', check_manager ) )
self._cog_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.cog, menu_items )
self._cog_button = ClientGUICommon.BetterBitmapButton( self, CC.GlobalBMPs.cog, self._ShowCogMenu )
self._cancel_button = ClientGUICommon.BetterBitmapButton( self, CC.GlobalBMPs.stop, self.Cancel )
#
@ -597,6 +579,30 @@ class NetworkJobControl( wx.Panel ):
self.SetSizer( hbox )
def _ShowCogMenu( self ):
menu = wx.Menu()
if self._network_job is not None:
if self._network_job.ObeysBandwidth():
ClientGUIMenus.AppendMenuItem( self, menu, 'override bandwidth rules for this job', 'Tell the current job to ignore existing bandwidth rules and go ahead anyway.', self._network_job.OverrideBandwidth )
if not self._network_job.TokensOK():
ClientGUIMenus.AppendMenuItem( self, menu, 'override gallery slot requirements for this job', 'Force-allow this download to proceed, ignoring the normal gallery wait times.', self._network_job.OverrideToken )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuCheckItem( self, menu, 'auto-override bandwidth rules for all jobs here after five seconds', 'Ignore existing bandwidth rules for all jobs under this control, instead waiting a flat five seconds.', self._auto_override_bandwidth_rules, self.FlipAutoOverrideBandwidth )
HG.client_controller.PopupMenu( self._cog_button, menu )
def _OverrideBandwidthIfAppropriate( self ):
if self._network_job is None or self._network_job.NoEngineYet():
@ -700,11 +706,6 @@ class NetworkJobControl( wx.Panel ):
def AutoOverrideBandwidth( self ):
return self._auto_override_bandwidth_rules
def Cancel( self ):
if self._network_job is not None:
@ -718,31 +719,11 @@ class NetworkJobControl( wx.Panel ):
self.SetNetworkJob( None )
def CurrentJobOverridesBandwidth( self ):
if self._network_job is None:
return None
else:
return not self._network_job.ObeysBandwidth()
def FlipAutoOverrideBandwidth( self ):
self._auto_override_bandwidth_rules = not self._auto_override_bandwidth_rules
def FlipOverrideBandwidthForCurrentJob( self ):
if self._network_job is not None:
self._network_job.OverrideBandwidth()
def SetNetworkJob( self, network_job ):
if network_job is None:

View File

@ -12,6 +12,7 @@ import ClientSerialisable
import ClientThreading
import HydrusConstants as HC
import HydrusData
import HydrusExceptions
import HydrusGlobals as HG
import HydrusPaths
import HydrusText
@ -50,7 +51,7 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
self.SetSizer( vbox )
self._list_ctrl.Bind( wx.EVT_RIGHT_DOWN, self.EventShowMenu )
self._list_ctrl.AddMenuCallable( self._GetListCtrlMenu )
self._controller.sub( self, 'NotifyFileSeedsUpdated', 'file_seed_cache_file_seeds_updated' )
@ -154,6 +155,42 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
def _GetListCtrlMenu( self ):
selected_file_seeds = self._list_ctrl.GetData( only_selected = True )
if len( selected_file_seeds ) == 0:
raise HydrusExceptions.DataMissing()
menu = wx.Menu()
can_show_files_in_new_page = True in ( file_seed.HasHash() for file_seed in selected_file_seeds )
if can_show_files_in_new_page:
ClientGUIMenus.AppendMenuItem( self, menu, 'open selected import files in a new page', 'Show all the known selected files in a new thumbnail page. This is complicated, so cannot always be guaranteed, even if the import says \'success\'.', self._ShowSelectionInNewPage )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'copy sources', 'Copy all the selected sources to clipboard.', self._CopySelectedFileSeedData )
ClientGUIMenus.AppendMenuItem( self, menu, 'copy notes', 'Copy all the selected notes to clipboard.', self._CopySelectedNotes )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'open sources', 'Open all the selected sources in your file explorer or web browser.', self._OpenSelectedFileSeedData )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'try again', 'Reset the progress of all the selected imports.', HydrusData.Call( self._SetSelected, CC.STATUS_UNKNOWN ) )
ClientGUIMenus.AppendMenuItem( self, menu, 'skip', 'Skip all the selected imports.', HydrusData.Call( self._SetSelected, CC.STATUS_SKIPPED ) )
ClientGUIMenus.AppendMenuItem( self, menu, 'delete from list', 'Remove all the selected imports.', self._DeleteSelected )
return menu
def _OpenSelectedFileSeedData( self ):
file_seeds = self._list_ctrl.GetData( only_selected = True )
@ -209,40 +246,6 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
self._file_seed_cache.NotifyFileSeedsUpdated( file_seeds )
def _ShowMenuIfNeeded( self ):
selected_file_seeds = self._list_ctrl.GetData( only_selected = True )
if len( selected_file_seeds ) > 0:
menu = wx.Menu()
can_show_files_in_new_page = True in ( file_seed.HasHash() for file_seed in selected_file_seeds )
if can_show_files_in_new_page:
ClientGUIMenus.AppendMenuItem( self, menu, 'open selected import files in a new page', 'Show all the known selected files in a new thumbnail page. This is complicated, so cannot always be guaranteed, even if the import says \'success\'.', self._ShowSelectionInNewPage )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'copy sources', 'Copy all the selected sources to clipboard.', self._CopySelectedFileSeedData )
ClientGUIMenus.AppendMenuItem( self, menu, 'copy notes', 'Copy all the selected notes to clipboard.', self._CopySelectedNotes )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'open sources', 'Open all the selected sources in your file explorer or web browser.', self._OpenSelectedFileSeedData )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'try again', 'Reset the progress of all the selected imports.', HydrusData.Call( self._SetSelected, CC.STATUS_UNKNOWN ) )
ClientGUIMenus.AppendMenuItem( self, menu, 'skip', 'Skip all the selected imports.', HydrusData.Call( self._SetSelected, CC.STATUS_SKIPPED ) )
ClientGUIMenus.AppendMenuItem( self, menu, 'delete from list', 'Remove all the selected imports.', self._DeleteSelected )
HG.client_controller.PopupMenu( self, menu )
def _ShowSelectionInNewPage( self ):
hashes = []
@ -314,13 +317,6 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
self.Layout()
def EventShowMenu( self, event ):
wx.CallAfter( self._ShowMenuIfNeeded )
event.Skip() # let the right click event go through before doing menu, in case selection should happen
def GetValue( self ):
return self._file_seed_cache

View File

@ -12,6 +12,7 @@ import ClientSerialisable
import ClientThreading
import HydrusConstants as HC
import HydrusData
import HydrusExceptions
import HydrusGlobals as HG
import HydrusPaths
import HydrusText
@ -52,7 +53,7 @@ class EditGallerySeedLogPanel( ClientGUIScrolledPanels.EditPanel ):
self.SetSizer( vbox )
self._list_ctrl.Bind( wx.EVT_RIGHT_DOWN, self.EventShowMenu )
self._list_ctrl.AddMenuCallable( self._GetListCtrlMenu )
self._controller.sub( self, 'NotifyGallerySeedsUpdated', 'gallery_seed_log_gallery_seeds_updated' )
@ -127,6 +128,41 @@ class EditGallerySeedLogPanel( ClientGUIScrolledPanels.EditPanel ):
def _GetListCtrlMenu( self ):
selected_gallery_seeds = self._list_ctrl.GetData( only_selected = True )
if len( selected_gallery_seeds ) == 0:
raise HydrusExceptions.DataMissing()
menu = wx.Menu()
ClientGUIMenus.AppendMenuItem( self, menu, 'copy urls', 'Copy all the selected urls to clipboard.', self._CopySelectedGalleryURLs )
ClientGUIMenus.AppendMenuItem( self, menu, 'copy notes', 'Copy all the selected notes to clipboard.', self._CopySelectedNotes )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'open urls', 'Open all the selected urls in your web browser.', self._OpenSelectedGalleryURLs )
ClientGUIMenus.AppendSeparator( menu )
if not self._read_only:
ClientGUIMenus.AppendMenuItem( self, menu, 'try again (just this one page)', 'Schedule this url to occur again.', HydrusData.Call( self._TrySelectedAgain, False ) )
if self._can_generate_more_pages:
ClientGUIMenus.AppendMenuItem( self, menu, 'try again (and allow search to continue)', 'Schedule this url to occur again and continue.', HydrusData.Call( self._TrySelectedAgain, True ) )
ClientGUIMenus.AppendMenuItem( self, menu, 'skip', 'Skip all the selected urls.', HydrusData.Call( self._SetSelected, CC.STATUS_SKIPPED ) )
return menu
def _OpenSelectedGalleryURLs( self ):
gallery_seeds = self._list_ctrl.GetData( only_selected = True )
@ -165,39 +201,6 @@ class EditGallerySeedLogPanel( ClientGUIScrolledPanels.EditPanel ):
self._gallery_seed_log.NotifyGallerySeedsUpdated( gallery_seeds )
def _ShowMenuIfNeeded( self ):
selected_gallery_seeds = self._list_ctrl.GetData( only_selected = True )
if len( selected_gallery_seeds ) > 0:
menu = wx.Menu()
ClientGUIMenus.AppendMenuItem( self, menu, 'copy urls', 'Copy all the selected urls to clipboard.', self._CopySelectedGalleryURLs )
ClientGUIMenus.AppendMenuItem( self, menu, 'copy notes', 'Copy all the selected notes to clipboard.', self._CopySelectedNotes )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'open urls', 'Open all the selected urls in your web browser.', self._OpenSelectedGalleryURLs )
ClientGUIMenus.AppendSeparator( menu )
if not self._read_only:
ClientGUIMenus.AppendMenuItem( self, menu, 'try again (just this one page)', 'Schedule this url to occur again.', HydrusData.Call( self._TrySelectedAgain, False ) )
if self._can_generate_more_pages:
ClientGUIMenus.AppendMenuItem( self, menu, 'try again (and allow search to continue)', 'Schedule this url to occur again and continue.', HydrusData.Call( self._TrySelectedAgain, True ) )
ClientGUIMenus.AppendMenuItem( self, menu, 'skip', 'Skip all the selected urls.', HydrusData.Call( self._SetSelected, CC.STATUS_SKIPPED ) )
HG.client_controller.PopupMenu( self, menu )
def _TrySelectedAgain( self, can_generate_more_pages ):
new_gallery_seeds = []
@ -266,13 +269,6 @@ class EditGallerySeedLogPanel( ClientGUIScrolledPanels.EditPanel ):
self.Layout()
def EventShowMenu( self, event ):
wx.CallAfter( self._ShowMenuIfNeeded )
event.Skip() # let the right click event go through before doing menu, in case selection should happen
def GetValue( self ):
return self._gallery_seed_log

View File

@ -2578,11 +2578,6 @@ class ListBoxTagsSelection( ListBoxTags ):
self._sort = HC.options[ 'default_tag_sort' ]
if not include_counts and self._sort in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
self._sort = CC.SORT_BY_LEXICOGRAPHIC_ASC
self._last_media = set()
self._tag_service_key = CC.COMBINED_TAG_SERVICE_KEY
@ -2724,130 +2719,14 @@ class ListBoxTagsSelection( ListBoxTags ):
def _SortTags( self ):
def lexicographic_key( term ):
tag = self._terms_to_texts[ term ]
( namespace, subtag ) = HydrusTags.SplitTag( tag )
comparable_subtag = HydrusTags.ConvertTagToSortable( subtag )
# 'cat' < 'character:rei'
# 'page:3' < 'page:20'
# '1' < 'series:eva'
# note that 'test' < ( 1, '' ) but u'test' > ( 1, '' ) wew
if namespace == '':
return ( comparable_subtag, comparable_subtag )
else:
return ( namespace, comparable_subtag )
tags_to_count = collections.Counter()
def incidence_key( term ):
return tags_to_count[ term ]
if self._show_current: tags_to_count.update( self._current_tags_to_count )
if self._show_deleted: tags_to_count.update( self._deleted_tags_to_count )
if self._show_pending: tags_to_count.update( self._pending_tags_to_count )
if self._show_petitioned: tags_to_count.update( self._petitioned_tags_to_count )
def namespace_key( term ):
tag = term
( namespace, subtag ) = HydrusTags.SplitTag( tag )
if namespace == '':
namespace = '{' # '{' is above 'z' in ascii, so this works for most situations
return namespace
def namespace_lexicographic_key( term ):
tag = term
# '{' is above 'z' in ascii, so this works for most situations
( namespace, subtag ) = HydrusTags.SplitTag( tag )
if namespace == '':
return ( '{', HydrusTags.ConvertTagToSortable( subtag ) )
else:
return ( namespace, HydrusTags.ConvertTagToSortable( subtag ) )
if self._sort in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
tags_to_count = collections.Counter()
if self._show_current: tags_to_count.update( self._current_tags_to_count )
if self._show_deleted: tags_to_count.update( self._deleted_tags_to_count )
if self._show_pending: tags_to_count.update( self._pending_tags_to_count )
if self._show_petitioned: tags_to_count.update( self._petitioned_tags_to_count )
# let's establish a-z here for equal incidence values later
if self._sort in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC ):
self._ordered_terms.sort( key = lexicographic_key, reverse = True )
reverse = False
elif self._sort in ( CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
self._ordered_terms.sort( key = lexicographic_key )
reverse = True
self._ordered_terms.sort( key = incidence_key, reverse = reverse )
if self._sort in ( CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
# python list sort is stable, so lets now sort again
if self._sort == CC.SORT_BY_INCIDENCE_NAMESPACE_ASC:
reverse = True
elif self._sort == CC.SORT_BY_INCIDENCE_NAMESPACE_DESC:
reverse = False
self._ordered_terms.sort( key = namespace_key, reverse = reverse )
else:
if self._sort in ( CC.SORT_BY_LEXICOGRAPHIC_DESC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
reverse = True
elif self._sort in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC ):
reverse = False
if self._sort in ( CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
key = namespace_lexicographic_key
elif self._sort in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_DESC ):
key = lexicographic_key
self._ordered_terms.sort( key = key, reverse = reverse )
ClientTags.SortTags( self._sort, self._ordered_terms, tags_to_count )
self._DataHasChanged()

View File

@ -512,6 +512,8 @@ class BetterListCtrl( wx.ListCtrl, ListCtrlAutoWidthMixin ):
self._data_to_tuples_func = data_to_tuples_func
self._menu_callable = None
self._sort_column = 0
self._sort_asc = True
@ -624,6 +626,20 @@ class BetterListCtrl( wx.ListCtrl, ListCtrlAutoWidthMixin ):
def _ShowMenu( self ):
try:
menu = self._menu_callable()
except HydrusExceptions.DataMissing:
return
HG.client_controller.PopupMenu( self, menu )
def _SortDataInfo( self ):
data_infos = list( self._indices_to_data_info.values() )
@ -698,6 +714,13 @@ class BetterListCtrl( wx.ListCtrl, ListCtrlAutoWidthMixin ):
wx.QueueEvent( self.GetEventHandler(), ListCtrlEvent( -1 ) )
def AddMenuCallable( self, menu_callable ):
self._menu_callable = menu_callable
self.Bind( wx.EVT_RIGHT_DOWN, self.EventShowMenu )
def DeleteDatas( self, datas ):
deletees = [ ( self._data_to_indices[ data ], data ) for data in datas ]
@ -824,6 +847,13 @@ class BetterListCtrl( wx.ListCtrl, ListCtrlAutoWidthMixin ):
def EventShowMenu( self, event ):
wx.CallAfter( self._ShowMenu )
event.Skip() # let the right click event go through before doing menu, in case selection should happen
def GetData( self, only_selected = False ):
if only_selected:
@ -1154,7 +1184,6 @@ class BetterListCtrlPanel( wx.Panel ):
dlg.ShowModal()
def _GetExportObject( self ):

View File

@ -949,7 +949,7 @@ class ManagementPanelDuplicateFilter( ManagementPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'duplicates.html' ) )
menu_items.append( ( 'normal', 'show some simpler help here', 'Throw up a message box with some simple help.', self._ShowSimpleHelp ) )
menu_items.append( ( 'normal', 'open the html duplicates help', 'Open the help page for duplicates processing in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the html duplicates help', 'Open the help page for duplicates processing in your web browser.', page_func ) )
self._help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -1642,6 +1642,8 @@ class ManagementPanelImporterMultipleGallery( ManagementPanelImporter ):
self._UpdateImportStatus()
self._gallery_importers_listctrl.AddMenuCallable( self._GetListCtrlMenu )
def _CanClearHighlight( self ):
@ -1772,6 +1774,41 @@ class ManagementPanelImporterMultipleGallery( ManagementPanelImporter ):
return ( display_tuple, sort_tuple )
def _CopySelectedQueries( self ):
gallery_importers = self._gallery_importers_listctrl.GetData( only_selected = True )
if len( gallery_importers ) > 0:
separator = os.linesep * 2
text = separator.join( ( gallery_importer.GetQueryText() for gallery_importer in gallery_importers ) )
HG.client_controller.pub( 'clipboard', 'text', text )
def _GetListCtrlMenu( self ):
selected_watchers = self._gallery_importers_listctrl.GetData( only_selected = True )
if len( selected_watchers ) == 0:
raise HydrusExceptions.DataMissing()
menu = wx.Menu()
ClientGUIMenus.AppendMenuItem( self, menu, 'copy queries', 'Copy all the selected downloaders\' queries to clipboard.', self._CopySelectedQueries )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'pause/play files', 'Pause/play all the selected downloaders\' file queues.', self._PausePlayFiles )
ClientGUIMenus.AppendMenuItem( self, menu, 'pause/play search', 'Pause/play all the selected downloaders\' gallery searches.', self._PausePlayGallery )
return menu
def _HighlightGalleryImport( self, new_highlight ):
if new_highlight == self._highlighted_gallery_import:
@ -2164,6 +2201,8 @@ class ManagementPanelImporterMultipleWatcher( ManagementPanelImporter ):
#
self._watchers_listctrl.AddMenuCallable( self._GetListCtrlMenu )
self._UpdateImportStatus()
HG.client_controller.sub( self, 'PendURL', 'pend_url' )
@ -2325,6 +2364,42 @@ class ManagementPanelImporterMultipleWatcher( ManagementPanelImporter ):
return ( display_tuple, sort_tuple )
def _CopySelectedURLs( self ):
watchers = self._watchers_listctrl.GetData( only_selected = True )
if len( watchers ) > 0:
separator = os.linesep * 2
text = separator.join( ( watcher.GetURL() for watcher in watchers ) )
HG.client_controller.pub( 'clipboard', 'text', text )
def _GetListCtrlMenu( self ):
selected_watchers = self._watchers_listctrl.GetData( only_selected = True )
if len( selected_watchers ) == 0:
raise HydrusExceptions.DataMissing()
menu = wx.Menu()
ClientGUIMenus.AppendMenuItem( self, menu, 'copy urls', 'Copy all the selected watchers\' urls to clipboard.', self._CopySelectedURLs )
ClientGUIMenus.AppendMenuItem( self, menu, 'open urls', 'Open all the selected watchers\' urls in your browser.', self._OpenSelectedURLs )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( self, menu, 'pause/play files', 'Pause/play all the selected watchers\' file queues.', self._PausePlayFiles )
ClientGUIMenus.AppendMenuItem( self, menu, 'pause/play checking', 'Pause/play all the selected watchers\' checking routines.', self._PausePlayChecking )
return menu
def _HighlightWatcher( self, new_highlight ):
if new_highlight == self._highlighted_watcher:
@ -2373,6 +2448,32 @@ class ManagementPanelImporterMultipleWatcher( ManagementPanelImporter ):
def _OpenSelectedURLs( self ):
watchers = self._watchers_listctrl.GetData( only_selected = True )
if len( watchers ) > 0:
if len( watchers ) > 10:
message = 'You have many watchers selected--are you sure you want to open them all?'
with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
if dlg.ShowModal() != wx.ID_YES:
return
for watcher in watchers:
ClientPaths.LaunchURLInWebBrowser( watcher.GetURL() )
def _OptionsUpdated( self, *args, **kwargs ):
self._multiple_watcher_import.SetOptions( self._checker_options.GetValue(), self._file_import_options.GetValue(), self._tag_import_options.GetValue() )

View File

@ -9,8 +9,10 @@ import ClientGUIControls
import ClientGUIListBoxes
import ClientGUIListCtrl
import ClientGUIScrolledPanels
import ClientGUIScrolledPanelsEdit
import ClientGUISerialisable
import ClientGUITopLevelWindows
import ClientNetworkingDomain
import ClientNetworkingJobs
import ClientParsing
import ClientPaths
@ -145,6 +147,345 @@ class StringMatchButton( ClientGUICommon.BetterButton ):
self._UpdateLabel()
class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
def __init__( self, parent, network_engine ):
ClientGUIScrolledPanels.ReviewPanel.__init__( self, parent )
self._network_engine = network_engine
menu_items = []
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'downloader_sharing.html' ) )
menu_items.append( ( 'normal', 'open the downloader sharing help', 'Open the help page for sharing downloaders in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
help_hbox = ClientGUICommon.WrapInText( help_button, self, 'help for this panel -->', wx.Colour( 0, 0, 255 ) )
listctrl_panel = ClientGUIListCtrl.BetterListCtrlPanel( self )
columns = [ ( 'name', -1 ), ( 'type', 40 ) ]
self._listctrl = ClientGUIListCtrl.BetterListCtrl( listctrl_panel, 'dowloader_export', 14, 36, columns, self._ConvertContentToListCtrlTuples, delete_key_callback = self._Delete )
self._listctrl.Sort( 1 )
listctrl_panel.SetListCtrl( self._listctrl )
listctrl_panel.AddButton( 'add gug', self._AddGUG )
listctrl_panel.AddButton( 'add url class', self._AddURLMatch )
listctrl_panel.AddButton( 'add parser', self._AddParser )
listctrl_panel.AddButton( 'delete', self._Delete, enabled_only_on_selection = True )
listctrl_panel.AddSeparator()
listctrl_panel.AddButton( 'export to png', self._Export, enabled_check_func = self._CanExport )
#
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.Add( help_hbox, CC.FLAGS_BUTTON_SIZER )
vbox.Add( listctrl_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
self.SetSizer( vbox )
def _AddGUG( self ):
choosable_gugs = [ gug for gug in self._network_engine.domain_manager.GetGUGs() if gug.IsFunctional() ]
for obj in self._listctrl.GetData():
if obj in choosable_gugs:
choosable_gugs.remove( obj )
choice_tuples = [ ( gug.GetName(), gug, False ) for gug in choosable_gugs ]
with ClientGUITopLevelWindows.DialogEdit( self, 'select gugs' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditChooseMultiple( dlg, choice_tuples )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
gugs_to_include = panel.GetValue()
else:
return
url_matches_to_include = self._GetURLMatchesToInclude( gugs_to_include )
url_matches_to_include = self._FlushOutURLMatchesWithAPILinks( url_matches_to_include )
parsers_to_include = self._GetParsersToInclude( url_matches_to_include )
self._listctrl.AddDatas( gugs_to_include )
self._listctrl.AddDatas( url_matches_to_include )
self._listctrl.AddDatas( parsers_to_include )
def _AddParser( self ):
choosable_parsers = list( self._network_engine.domain_manager.GetParsers() )
for obj in self._listctrl.GetData():
if obj in choosable_parsers:
choosable_parsers.remove( obj )
choice_tuples = [ ( parser.GetName(), parser, False ) for parser in choosable_parsers ]
with ClientGUITopLevelWindows.DialogEdit( self, 'select parsers' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditChooseMultiple( dlg, choice_tuples )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
parsers_to_include = panel.GetValue()
else:
return
self._listctrl.AddDatas( parsers_to_include )
def _AddURLMatch( self ):
choosable_url_matches = list( self._network_engine.domain_manager.GetURLMatches() )
for obj in self._listctrl.GetData():
if obj in choosable_url_matches:
choosable_url_matches.remove( obj )
choice_tuples = [ ( url_match.GetName(), url_match, False ) for url_match in choosable_url_matches ]
with ClientGUITopLevelWindows.DialogEdit( self, 'select url classes' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditChooseMultiple( dlg, choice_tuples )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
url_matches_to_include = panel.GetValue()
url_matches_to_include = self._FlushOutURLMatchesWithAPILinks( url_matches_to_include )
parsers_to_include = self._GetParsersToInclude( url_matches_to_include )
self._listctrl.AddDatas( url_matches_to_include )
self._listctrl.AddDatas( parsers_to_include )
def _CanExport( self ):
return len( self._listctrl.GetData() ) > 0
def _ConvertContentToListCtrlTuples( self, content ):
name = content.GetName()
t = content.SERIALISABLE_NAME
pretty_name = name
pretty_t = t
display_tuple = ( pretty_name, pretty_t )
sort_tuple = ( name, t )
return ( display_tuple, sort_tuple )
def _Delete( self ):
with ClientGUIDialogs.DialogYesNo( self, 'Remove all selected?' ) as dlg:
if dlg.ShowModal() == wx.ID_YES:
self._listctrl.DeleteSelected()
def _Export( self ):
export_object = HydrusSerialisable.SerialisableList( self._listctrl.GetData() )
message = 'The end-user will see this sort of summary:'
message += os.linesep * 2
message += os.linesep.join( ( obj.GetSafeSummary() for obj in export_object[:20] ) )
if len( export_object ) > 20:
message += os.linesep
message += '(and ' + HydrusData.ToHumanInt( len( export_object ) - 20 ) + ' others)'
message += os.linesep * 2
message += 'Does that look good? (Ideally, every object should have correct and sane domains listed here)'
with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
if dlg.ShowModal() != wx.ID_YES:
return
gug_names = set()
for obj in export_object:
if isinstance( obj, ( ClientNetworkingDomain.GalleryURLGenerator, ClientNetworkingDomain.NestedGalleryURLGenerator ) ):
gug_names.add( obj.GetName() )
gug_names = list( gug_names )
gug_names.sort()
num_gugs = len( gug_names )
with ClientGUITopLevelWindows.DialogNullipotent( self, 'export to png' ) as dlg:
title = 'easy-import downloader png'
if num_gugs == 0:
description = 'some download components'
else:
title += ' - ' + HydrusData.ToHumanInt( num_gugs ) + ' downloaders'
description = ', '.join( gug_names )
panel = ClientGUISerialisable.PngExportPanel( dlg, export_object, title = title, description = description )
dlg.SetPanel( panel )
dlg.ShowModal()
def _FlushOutURLMatchesWithAPILinks( self, url_matches ):
url_matches_to_include = set( url_matches )
api_links_dict = dict( ClientNetworkingDomain.ConvertURLMatchesIntoAPIPairs( self._network_engine.domain_manager.GetURLMatches() ) )
for url_match in url_matches:
added_this_cycle = set()
while url_match in api_links_dict and url_match not in added_this_cycle:
added_this_cycle.add( url_match )
url_match = api_links_dict[ url_match ]
url_matches_to_include.add( url_match )
return list( url_matches_to_include )
def _GetParsersToInclude( self, url_matches ):
parsers_to_include = set()
for url_match in url_matches:
example_url = url_match.GetExampleURL()
( url_type, match_name, can_parse ) = self._network_engine.domain_manager.GetURLParseCapability( example_url )
if can_parse:
try:
( url_to_fetch, parser ) = self._network_engine.domain_manager.GetURLToFetchAndParser( example_url )
parsers_to_include.add( parser )
except:
pass
return list( parsers_to_include )
def _GetURLMatchesToInclude( self, gugs ):
url_matches_to_include = set()
for gug in gugs:
if isinstance( gug, ClientNetworkingDomain.GalleryURLGenerator ):
example_urls = ( gug.GetExampleURL(), )
elif isinstance( gug, ClientNetworkingDomain.NestedGalleryURLGenerator ):
example_urls = gug.GetExampleURLs()
for example_url in example_urls:
url_match = self._network_engine.domain_manager.GetURLMatch( example_url )
if url_match is not None:
url_matches_to_include.add( url_match )
# add post url matches from same domain
domain = ClientNetworkingDomain.ConvertURLIntoSecondLevelDomain( example_url )
for um in list( self._network_engine.domain_manager.GetURLMatches() ):
if ClientNetworkingDomain.ConvertURLIntoSecondLevelDomain( um.GetExampleURL() ) == domain and um.GetURLType() in ( HC.URL_TYPE_POST, HC.URL_TYPE_FILE ):
url_matches_to_include.add( um )
return list( url_matches_to_include )
class EditCompoundFormulaPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent, formula, test_context ):
@ -157,7 +498,7 @@ class EditCompoundFormulaPanel( ClientGUIScrolledPanels.EditPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'downloader_parsers_formulae.html#compound_formula' ) )
menu_items.append( ( 'normal', 'open the compound formula help', 'Open the help page for compound formulae in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the compound formula help', 'Open the help page for compound formulae in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -387,7 +728,7 @@ class EditContextVariableFormulaPanel( ClientGUIScrolledPanels.EditPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'downloader_parsers_formulae.html#context_variable_formula' ) )
menu_items.append( ( 'normal', 'open the context variable formula help', 'Open the help page for context variable formulae in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the context variable formula help', 'Open the help page for context variable formulae in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -842,7 +1183,7 @@ class EditHTMLFormulaPanel( ClientGUIScrolledPanels.EditPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'downloader_parsers_formulae.html#html_formula' ) )
menu_items.append( ( 'normal', 'open the html formula help', 'Open the help page for html formulae in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the html formula help', 'Open the help page for html formulae in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -1210,7 +1551,7 @@ class EditJSONFormulaPanel( ClientGUIScrolledPanels.EditPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'downloader_parsers_formulae.html#json_formula' ) )
menu_items.append( ( 'normal', 'open the json formula help', 'Open the help page for json formulae in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the json formula help', 'Open the help page for json formulae in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -1440,7 +1781,7 @@ class EditContentParserPanel( ClientGUIScrolledPanels.EditPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'downloader_parsers_content_parsers.html#content_parsers' ) )
menu_items.append( ( 'normal', 'open the content parsers help', 'Open the help page for content parsers in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the content parsers help', 'Open the help page for content parsers in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -2454,7 +2795,7 @@ class EditPageParserPanel( ClientGUIScrolledPanels.EditPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'downloader_parsers_page_parsers.html#page_parsers' ) )
menu_items.append( ( 'normal', 'open the page parser help', 'Open the help page for page parsers in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the page parser help', 'Open the help page for page parsers in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )

View File

@ -1863,7 +1863,7 @@ class EditGUGsPanel( ClientGUIScrolledPanels.EditPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'downloader_gugs.html' ) )
menu_items.append( ( 'normal', 'open the gugs help', 'Open the help page for gugs in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the gugs help', 'Open the help page for gugs in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -3849,7 +3849,7 @@ class EditSubscriptionsPanel( ClientGUIScrolledPanels.EditPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'getting_started_subscriptions.html' ) )
menu_items.append( ( 'normal', 'open the html subscriptions help', 'Open the help page for subscriptions in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the html subscriptions help', 'Open the help page for subscriptions in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -3901,6 +3901,8 @@ class EditSubscriptionsPanel( ClientGUIScrolledPanels.EditPanel ):
self._subscriptions.AddDatas( subscriptions )
self._subscriptions.Sort( 0 )
#
vbox = wx.BoxSizer( wx.VERTICAL )
@ -6042,7 +6044,7 @@ class EditURLMatchesPanel( ClientGUIScrolledPanels.EditPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'downloader_url_classes.html' ) )
menu_items.append( ( 'normal', 'open the url classes help', 'Open the help page for url classes in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the url classes help', 'Open the help page for url classes in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -6120,7 +6122,7 @@ class EditURLMatchesPanel( ClientGUIScrolledPanels.EditPanel ):
HydrusSerialisable.SetNonDupeName( url_match, self._GetExistingNames() )
url_match.RegenMatchKey()
url_match.RegenerateMatchKey()
self._list_ctrl.AddDatas( ( url_match, ) )
@ -6287,6 +6289,8 @@ class EditURLMatchLinksPanel( ClientGUIScrolledPanels.EditPanel ):
self._api_pairs_list_ctrl.AddDatas( api_pairs )
self._api_pairs_list_ctrl.Sort( 0 )
# anything that goes to an api url will be parsed by that api's parser--it can't have its own
api_pair_unparsable_url_matches = set()

View File

@ -1,6 +1,7 @@
import ClientConstants as CC
import ClientData
import ClientDefaults
import ClientDragDrop
import ClientExporting
import ClientGUICommon
import ClientGUIControls
@ -18,8 +19,10 @@ import ClientGUITopLevelWindows
import ClientNetworking
import ClientNetworkingContexts
import ClientNetworkingDomain
import ClientParsing
import ClientPaths
import ClientRendering
import ClientSerialisable
import ClientTags
import ClientThreading
import collections
@ -333,7 +336,7 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'database_migration.html' ) )
menu_items.append( ( 'normal', 'open the html migration help', 'Open the help page for database migration in your web browesr.', page_func ) )
menu_items.append( ( 'normal', 'open the html migration help', 'Open the help page for database migration in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
@ -1496,6 +1499,276 @@ class ReviewAllBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
def __init__( self, parent, network_engine ):
ClientGUIScrolledPanels.ReviewPanel.__init__( self, parent )
self._network_engine = network_engine
vbox = wx.BoxSizer( wx.VERTICAL )
menu_items = []
page_func = HydrusData.Call( ClientPaths.LaunchPathInWebBrowser, os.path.join( HC.HELP_DIR, 'adding_new_downloaders.html' ) )
menu_items.append( ( 'normal', 'open the easy downloader import help', 'Open the help page for easily importing downloaders in your web browser.', page_func ) )
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
help_hbox = ClientGUICommon.WrapInText( help_button, self, 'help for this panel -->', wx.Colour( 0, 0, 255 ) )
st = ClientGUICommon.BetterStaticText( self, label = 'Drop downloader-encoded pngs onto Lain to import.' )
lain_path = os.path.join( HC.STATIC_DIR, 'lain.jpg' )
lain_bmp = ClientRendering.GenerateHydrusBitmap( lain_path, HC.IMAGE_JPEG ).GetWxBitmap()
win = ClientGUICommon.BufferedWindowIcon( self, lain_bmp )
self._select_from_list = wx.CheckBox( self )
if HG.client_controller.new_options.GetBoolean( 'advanced_mode' ):
self._select_from_list.SetValue( True )
vbox.Add( help_hbox, CC.FLAGS_BUTTON_SIZER )
vbox.Add( st, CC.FLAGS_CENTER )
vbox.Add( win, CC.FLAGS_CENTER )
vbox.Add( ClientGUICommon.WrapInText( self._select_from_list, self, 'select objects from list' ), CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
self.SetSizer( vbox )
#
win.SetDropTarget( ClientDragDrop.FileDropTarget( self, filenames_callable = self.ImportFromDragDrop ) )
def ImportFromDragDrop( self, paths ):
gugs = []
url_matches = []
parsers = []
num_misc_objects = 0
domain_manager = self._network_engine.domain_manager
for path in paths:
path = HydrusData.ToUnicode( path )
try:
payload = ClientSerialisable.LoadFromPng( path )
except Exception as e:
wx.MessageBox( HydrusData.ToUnicode( e ) )
return
try:
obj_list = HydrusSerialisable.CreateFromNetworkString( payload )
except:
wx.MessageBox( 'I could not understand what was encoded in the file ' + path + '!' )
continue
if not isinstance( obj_list, HydrusSerialisable.SerialisableList ):
wx.MessageBox( 'Unfortunately, ' + path + ' did not look like a package of download data! Instead, it looked like: ' + obj_list.SERIALISABLE_NAME )
continue
for obj in obj_list:
if isinstance( obj, ( ClientNetworkingDomain.GalleryURLGenerator, ClientNetworkingDomain.NestedGalleryURLGenerator ) ):
gugs.append( obj )
elif isinstance( obj, ClientNetworkingDomain.URLMatch ):
url_matches.append( obj )
elif isinstance( obj, ClientParsing.PageParser ):
parsers.append( obj )
else:
num_misc_objects += 1
if len( gugs ) + len( url_matches ) + len( parsers ) == 0:
if len( num_misc_objects ) > 0:
wx.MessageBox( 'I found ' + HydrusData.ToHumanInt( num_misc_objects ) + ' misc objects in that png, but nothing downloader related.' )
return
# url matches first
dupe_url_matches = []
num_exact_dupe_url_matches = 0
new_url_matches = []
for url_match in url_matches:
if domain_manager.AlreadyHaveExactlyThisURLMatch( url_match ):
dupe_url_matches.append( url_match )
num_exact_dupe_url_matches += 1
else:
new_url_matches.append( url_match )
# now gugs
num_exact_dupe_gugs = 0
new_gugs = []
for gug in gugs:
if domain_manager.AlreadyHaveExactlyThisGUG( gug ):
num_exact_dupe_gugs += 1
else:
new_gugs.append( gug )
#
# now parsers
num_exact_dupe_parsers = 0
new_parsers = []
for parser in parsers:
if domain_manager.AlreadyHaveExactlyThisParser( parser ):
num_exact_dupe_parsers += 1
else:
new_parsers.append( parser )
total_num_dupes = num_exact_dupe_gugs + num_exact_dupe_url_matches + num_exact_dupe_parsers
if len( new_gugs ) + len( new_url_matches ) + len( new_parsers ) == 0:
wx.MessageBox( 'All ' + HydrusData.ToHumanInt( total_num_dupes ) + ' downloader objects in that package appeared to already be in the client, so nothing need be added.' )
return
# let's start selecting what we want
if self._select_from_list.GetValue() == True:
choice_tuples = []
choice_tuples.extend( [ ( 'GUG: ' + gug.GetName(), gug, True ) for gug in new_gugs ] )
choice_tuples.extend( [ ( 'URL Class: ' + url_match.GetName(), url_match, True ) for url_match in new_url_matches ] )
choice_tuples.extend( [ ( 'Parser: ' + parser.GetName(), parser, True ) for parser in new_parsers ] )
with ClientGUITopLevelWindows.DialogEdit( self, 'select objects to add' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditChooseMultiple( dlg, choice_tuples )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
new_objects = panel.GetValue()
new_gugs = [ obj for obj in new_objects if isinstance( obj, ( ClientNetworkingDomain.GalleryURLGenerator, ClientNetworkingDomain.NestedGalleryURLGenerator ) ) ]
new_url_matches = [ obj for obj in new_objects if isinstance( obj, ClientNetworkingDomain.URLMatch ) ]
new_parsers = [ obj for obj in new_objects if isinstance( obj, ClientParsing.PageParser ) ]
else:
return
# final ask
new_gugs.sort( key = lambda o: o.GetName() )
new_url_matches.sort( key = lambda o: o.GetName() )
new_parsers.sort( key = lambda o: o.GetName() )
all_to_add = list( new_gugs )
all_to_add.extend( new_url_matches )
all_to_add.extend( new_parsers )
message = 'The client is about to add and link these objects:'
message += os.linesep * 2
message += os.linesep.join( ( obj.GetSafeSummary() for obj in all_to_add[:20] ) )
if len( all_to_add ) > 20:
message += os.linesep
message += '(and ' + HydrusData.ToHumanInt( len( all_to_add ) - 20 ) + ' others)'
message += os.linesep * 2
message += 'Does that sound good?'
with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
if dlg.ShowModal() != wx.ID_YES:
return
# ok, do it
if len( new_gugs ) > 0:
domain_manager.AddGUGs( new_gugs )
domain_manager.AutoAddURLMatchesAndParsers( new_url_matches, dupe_url_matches, new_parsers )
num_new_gugs = len( new_gugs )
num_aux = len( new_url_matches ) + len( new_parsers )
final_message = 'Successfully added ' + HydrusData.ToHumanInt( len( new_gugs ) ) + ' new downloaders and ' + HydrusData.ToHumanInt( len( new_url_matches ) + len( new_parsers ) ) + ' auxiliary objects.'
if total_num_dupes > 0:
final_message += ' ' + HydrusData.ToHumanInt( total_num_dupes ) + ' duplicate objects were not added (but some additional metadata may have been merged).'
wx.MessageBox( final_message )
class ReviewExportFilesPanel( ClientGUIScrolledPanels.ReviewPanel ):
def __init__( self, parent, flat_media ):
@ -1928,22 +2201,33 @@ class ReviewHowBonedAmI( ClientGUIScrolledPanels.ReviewPanel ):
vbox.Add( win, CC.FLAGS_CENTER )
num_archive_percent = float( num_archive ) / num_total
size_archive_percent = float( size_archive ) / size_total
num_inbox_percent = float( num_inbox ) / num_total
size_inbox_percent = float( size_inbox ) / size_total
archive_label = 'Archive: ' + HydrusData.ToHumanInt( num_archive ) + ' files (' + ClientData.ConvertZoomToPercentage( num_archive_percent ) + '), totalling ' + HydrusData.ConvertIntToBytes( size_archive ) + '(' + ClientData.ConvertZoomToPercentage( size_archive_percent ) + ')'
archive_st = ClientGUICommon.BetterStaticText( self, label = archive_label )
inbox_label = 'Inbox: ' + HydrusData.ToHumanInt( num_inbox ) + ' files (' + ClientData.ConvertZoomToPercentage( num_inbox_percent ) + '), totalling ' + HydrusData.ConvertIntToBytes( size_inbox ) + '(' + ClientData.ConvertZoomToPercentage( size_inbox_percent ) + ')'
inbox_st = ClientGUICommon.BetterStaticText( self, label = inbox_label )
vbox.Add( archive_st, CC.FLAGS_CENTER )
vbox.Add( inbox_st, CC.FLAGS_CENTER )
if num_total == 0:
nothing_label = 'You have yet to board the ride. Why don\'t you try importing some files? :^)'
nothing_st = ClientGUICommon.BetterStaticText( self, label = nothing_label )
vbox.Add( nothing_st, CC.FLAGS_CENTER )
else:
num_archive_percent = float( num_archive ) / num_total
size_archive_percent = float( size_archive ) / size_total
num_inbox_percent = float( num_inbox ) / num_total
size_inbox_percent = float( size_inbox ) / size_total
archive_label = 'Archive: ' + HydrusData.ToHumanInt( num_archive ) + ' files (' + ClientData.ConvertZoomToPercentage( num_archive_percent ) + '), totalling ' + HydrusData.ConvertIntToBytes( size_archive ) + '(' + ClientData.ConvertZoomToPercentage( size_archive_percent ) + ')'
archive_st = ClientGUICommon.BetterStaticText( self, label = archive_label )
inbox_label = 'Inbox: ' + HydrusData.ToHumanInt( num_inbox ) + ' files (' + ClientData.ConvertZoomToPercentage( num_inbox_percent ) + '), totalling ' + HydrusData.ConvertIntToBytes( size_inbox ) + '(' + ClientData.ConvertZoomToPercentage( size_inbox_percent ) + ')'
inbox_st = ClientGUICommon.BetterStaticText( self, label = inbox_label )
vbox.Add( archive_st, CC.FLAGS_CENTER )
vbox.Add( inbox_st, CC.FLAGS_CENTER )
self.SetSizer( vbox )

View File

@ -13,7 +13,7 @@ import wx
class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
def __init__( self, parent, payload_obj ):
def __init__( self, parent, payload_obj, title = None, description = None, payload_description = None ):
ClientGUIScrolledPanels.ReviewPanel.__init__( self, parent )
@ -37,7 +37,16 @@ class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
#
( payload_description, payload_string ) = ClientSerialisable.GetPayloadDescriptionAndString( self._payload_obj )
if payload_description is None:
( payload_description, payload_string ) = ClientSerialisable.GetPayloadDescriptionAndString( self._payload_obj )
else:
payload_string = ClientSerialisable.GetPayloadString( self._payload_obj )
payload_description += ' - ' + HydrusData.ConvertIntToBytes( len( payload_string ) )
self._payload_description.SetValue( payload_description )
@ -47,7 +56,11 @@ class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
last_png_export_dir = HG.client_controller.new_options.GetNoneableString( 'last_png_export_dir' )
if isinstance( self._payload_obj, HydrusSerialisable.SerialisableBaseNamed ):
if title is not None:
name = title
elif isinstance( self._payload_obj, HydrusSerialisable.SerialisableBaseNamed ):
name = self._payload_obj.GetName()
@ -58,6 +71,11 @@ class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
self._title.SetValue( name )
if description is not None:
self._text.SetValue( description )
if last_png_export_dir is not None:
filename = name + '.png'
@ -127,7 +145,8 @@ class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
width = self._width.GetValue()
( payload_description, payload_string ) = ClientSerialisable.GetPayloadDescriptionAndString( self._payload_obj )
payload_description = self._payload_description.GetValue()
payload_string = ClientSerialisable.GetPayloadString( self._payload_obj )
title = self._title.GetValue()
text = self._text.GetValue()

View File

@ -93,9 +93,9 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
#
self._redundant_st = ClientGUICommon.BetterStaticText( self, '' )
self._redundant_st = ClientGUICommon.BetterStaticText( self, '', style = wx.ST_ELLIPSIZE_END )
self._current_filter_st = ClientGUICommon.BetterStaticText( self, 'currently keeping: ' )
self._current_filter_st = ClientGUICommon.BetterStaticText( self, 'currently keeping: ', style = wx.ST_ELLIPSIZE_END )
#

View File

@ -666,8 +666,9 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
work_pending = self._file_seed_cache.WorkToDo() and not self._files_paused
no_delays = HydrusData.TimeHasPassed( self._no_work_until )
page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_pending and no_delays and page_shown
ok_to_work = work_pending and no_delays and page_shown and network_engine_good
while ok_to_work:
@ -695,8 +696,9 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
work_pending = self._file_seed_cache.WorkToDo() and not self._files_paused
no_delays = HydrusData.TimeHasPassed( self._no_work_until )
page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_pending and no_delays and page_shown
ok_to_work = work_pending and no_delays and page_shown and network_engine_good
@ -715,8 +717,9 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
work_pending = self._gallery_seed_log.WorkToDo() and not self._gallery_paused
no_delays = HydrusData.TimeHasPassed( self._no_work_until )
page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_pending and no_delays and page_shown
ok_to_work = work_pending and no_delays and page_shown and network_engine_good
while ok_to_work:
@ -746,8 +749,9 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
work_pending = self._gallery_seed_log.WorkToDo() and not self._gallery_paused
no_delays = HydrusData.TimeHasPassed( self._no_work_until )
page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_pending and no_delays and page_shown
ok_to_work = work_pending and no_delays and page_shown and network_engine_good

View File

@ -536,9 +536,12 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
work_to_do = self._file_seed_cache.WorkToDo() and not ( self._files_paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_to_do and network_engine_good
while work_to_do:
while ok_to_work:
try:
@ -561,6 +564,9 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
work_to_do = self._file_seed_cache.WorkToDo() and not ( self._files_paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_to_do and network_engine_good
@ -576,7 +582,11 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
return
ok_to_work = not ( self._queue_paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
queue_good = not self._queue_paused
page_shown = not HG.client_controller.PageClosedButNotDestroyed( page_key )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = queue_good and page_shown and network_engine_good
while ok_to_work:
@ -610,7 +620,11 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
return
ok_to_work = not ( self._queue_paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
queue_good = not self._queue_paused
page_shown = not HG.client_controller.PageClosedButNotDestroyed( page_key )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = queue_good and page_shown and network_engine_good
@ -992,9 +1006,12 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
work_to_do = self._file_seed_cache.WorkToDo() and not ( self._paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_to_do and network_engine_good
while work_to_do:
while ok_to_work:
try:
@ -1017,6 +1034,9 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
work_to_do = self._file_seed_cache.WorkToDo() and not ( self._paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_to_do and network_engine_good
@ -1033,9 +1053,12 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
work_to_do = self._gallery_seed_log.WorkToDo() and not ( self._paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_to_do and network_engine_good
while work_to_do:
while ok_to_work:
try:
@ -1058,6 +1081,9 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
work_to_do = self._gallery_seed_log.WorkToDo() and not ( self._paused or HG.client_controller.PageClosedButNotDestroyed( page_key ) )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_to_do and network_engine_good

View File

@ -1182,8 +1182,6 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
self._checker_options = checker_options
self._checking_paused = False
self._UpdateNextCheckTime()
self._UpdateFileVelocityStatus()
@ -1255,8 +1253,9 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
work_pending = self._file_seed_cache.WorkToDo() and not self._files_paused
no_delays = HydrusData.TimeHasPassed( self._no_work_until )
page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_pending and no_delays and page_shown
ok_to_work = work_pending and no_delays and page_shown and network_engine_good
while ok_to_work:
@ -1284,8 +1283,9 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
work_pending = self._file_seed_cache.WorkToDo() and not self._files_paused
no_delays = HydrusData.TimeHasPassed( self._no_work_until )
page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
ok_to_work = work_pending and no_delays and page_shown
ok_to_work = work_pending and no_delays and page_shown and network_engine_good
@ -1305,8 +1305,9 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
check_due = HydrusData.TimeHasPassed( self._next_check_time )
no_delays = HydrusData.TimeHasPassed( self._no_work_until )
page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
network_engine_good = not HG.client_controller.network_engine.IsBusy()
time_to_check = able_to_check and check_due and no_delays and page_shown
time_to_check = able_to_check and check_due and no_delays and page_shown and network_engine_good
if time_to_check:

View File

@ -112,6 +112,14 @@ class NetworkEngine( object ):
def IsBusy( self ):
with self._lock:
return len( self._jobs_awaiting_validity ) + len( self._jobs_awaiting_bandwidth ) + len( self._jobs_awaiting_login ) + len( self._jobs_awaiting_slot ) + len( self._jobs_running ) > 50
def IsRunning( self ):
with self._lock:

View File

@ -148,6 +148,10 @@ def ConvertQueryTextToDict( query_text ):
def ConvertURLMatchesIntoAPIPairs( url_matches ):
url_matches = list( url_matches )
NetworkDomainManager.STATICSortURLMatchesDescendingComplexity( url_matches )
pairs = []
for url_match in url_matches:
@ -170,6 +174,8 @@ def ConvertURLMatchesIntoAPIPairs( url_matches ):
pairs.append( ( url_match, other_url_match ) )
break
@ -742,7 +748,12 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
gugs = list( self._gugs )
gugs.extend( new_gugs )
for gug in new_gugs:
gug.SetNonDupeName( [ g.GetName() for g in gugs ] )
gugs.append( gug )
self.SetGUGs( gugs )
@ -754,12 +765,212 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
parsers = list( self._parsers )
parsers.extend( new_parsers )
for parser in new_parsers:
parser.SetNonDupeName( [ p.GetName() for p in parsers ] )
parsers.append( parser )
self.SetParsers( parsers )
def AddURLMatches( self, new_url_matches ):
with self._lock:
url_matches = list( self._url_matches )
for url_match in new_url_matches:
url_match.SetNonDupeName( [ u.GetName() for u in url_matches ] )
url_matches.append( url_match )
self.SetURLMatches( url_matches )
def AlreadyHaveExactlyThisGUG( self, new_gug ):
with self._lock:
# absent irrelevant variables, do we have the exact same object already in?
gug_key_and_name = new_gug.GetGUGKeyAndName()
dupe_gugs = [ gug.Duplicate() for gug in self._gugs ]
for dupe_gug in dupe_gugs:
dupe_gug.SetGUGKeyAndName( gug_key_and_name )
if dupe_gug.DumpToString() == new_gug.DumpToString():
return True
return False
def AlreadyHaveExactlyThisParser( self, new_parser ):
with self._lock:
# absent irrelevant variables, do we have the exact same object already in?
new_name = new_parser.GetName()
new_parser_key = new_parser.GetParserKey()
new_example_urls = new_parser.GetExampleURLs()
new_example_parsing_context = new_parser.GetExampleParsingContext()
dupe_parsers = [ ( parser.Duplicate(), parser ) for parser in self._parsers ]
for ( dupe_parser, parser ) in dupe_parsers:
dupe_parser.SetName( new_name )
dupe_parser.SetParserKey( new_parser_key )
dupe_parser.SetExampleURLs( new_example_urls )
dupe_parser.SetExampleParsingContext( new_example_parsing_context )
if dupe_parser.DumpToString() == new_parser.DumpToString():
# since these are the 'same', let's merge example urls
parser_example_urls = set( parser.GetExampleURLs() )
parser_example_urls.update( new_example_urls )
parser_example_urls = list( parser_example_urls )
parser.SetExampleURLs( parser_example_urls )
self._SetDirty()
return True
return False
def AlreadyHaveExactlyThisURLMatch( self, new_url_match ):
with self._lock:
# absent irrelevant variables, do we have the exact same object already in?
name = new_url_match.GetName()
match_key = new_url_match.GetMatchKey()
example_url = new_url_match.GetExampleURL()
dupe_url_matches = [ url_match.Duplicate() for url_match in self._url_matches ]
for dupe_url_match in dupe_url_matches:
dupe_url_match.SetName( name )
dupe_url_match.SetMatchKey( match_key )
dupe_url_match.SetExampleURL( example_url )
if dupe_url_match.DumpToString() == new_url_match.DumpToString():
return True
return False
def AutoAddURLMatchesAndParsers( self, new_url_matches, dupe_url_matches, new_parsers ):
for url_match in new_url_matches:
url_match.RegenerateMatchKey()
for parser in new_parsers:
parser.RegenerateParserKey()
# any existing url matches that already do the job of the new ones should be hung on to but renamed
with self._lock:
prefix = 'zzz - renamed due to auto-import - '
renamees = []
for existing_url_match in self._url_matches:
if existing_url_match.GetName().startswith( prefix ):
continue
for new_url_match in new_url_matches:
if new_url_match.Matches( existing_url_match.GetExampleURL() ) and existing_url_match.Matches( new_url_match.GetExampleURL ):
# the url matches match each other, so they are doing the same job
renamees.append( existing_url_match )
break
for renamee in renamees:
existing_names = [ url_match.GetName() for url_match in self._url_matches if url_match != renamee ]
renamee.SetName( prefix + renamee.GetName() )
renamee.SetNonDupeName( existing_names )
self.AddURLMatches( new_url_matches )
self.AddParsers( new_parsers )
# we want to match these url matches and parsers together if possible
with self._lock:
url_matches_to_link = list( new_url_matches )
# if downloader adds existing url match but updated parser, we want to update the existing link
for dupe_url_match in dupe_url_matches:
# this is to make sure we have the right match keys for the link update in a minute
actual_existing_dupe_url_match = self._GetURLMatch( dupe_url_match.GetExampleURL() )
if actual_existing_dupe_url_match is not None:
url_matches_to_link.append( actual_existing_dupe_url_match )
new_url_match_keys_to_parser_keys = NetworkDomainManager.STATICLinkURLMatchesAndParsers( url_matches_to_link, new_parsers, {} )
self._url_match_keys_to_parser_keys.update( new_url_match_keys_to_parser_keys )
# let's do a trytolink just in case there are loose ends due to some dupe being discarded earlier (e.g. url match is new, but parser was not).
self.TryToLinkURLMatchesAndParsers()
def CanValidateInPopup( self, network_contexts ):
# we can always do this for headers
@ -1202,7 +1413,7 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
for url_match in default_url_matches:
url_match.RegenMatchKey()
url_match.RegenerateMatchKey()
existing_url_matches = list( self._url_matches )
@ -1774,8 +1985,9 @@ class GalleryURLGenerator( HydrusSerialisable.SerialisableBaseNamed ):
# encode this gubbins since requests won't be able to do it
# this basically fixes e621 searches for 'male/female', which through some httpconf trickery are embedded in path but end up in a query, so need to be encoded right beforehand
# we need ToByteString as urllib.quote can't handle unicode hiragana etc...
search_terms = [ urllib.quote( search_term, safe = '' ) for search_term in search_terms ]
search_terms = [ urllib.quote( HydrusData.ToByteString( search_term ), safe = '' ) for search_term in search_terms ]
try:
@ -1817,11 +2029,24 @@ class GalleryURLGenerator( HydrusSerialisable.SerialisableBaseNamed ):
return self._initial_search_text
def GetSafeSummary( self ):
return 'Downloader "' + self._name + '" - ' + ConvertURLIntoDomain( self.GetExampleURL() )
def GetURLTemplateVariables( self ):
return ( self._url_template, self._replacement_phrase, self._search_terms_separator, self._example_search_text )
def SetGUGKeyAndName( self, gug_key_and_name ):
( gug_key, name ) = gug_key_and_name
self._gallery_url_generator_key = gug_key
self._name = name
def IsFunctional( self ):
try:
@ -1908,6 +2133,23 @@ class NestedGalleryURLGenerator( HydrusSerialisable.SerialisableBaseNamed ):
return gallery_urls
def GetExampleURLs( self ):
example_urls = []
for gug_key_and_name in self._gug_keys_and_names:
gug = HG.client_controller.network_engine.domain_manager.GetGUG( gug_key_and_name )
if gug is not None:
example_urls.append( gug.GetExampleURL() )
return example_urls
def GetGUGKey( self ):
return self._gallery_url_generator_key
@ -1938,6 +2180,11 @@ class NestedGalleryURLGenerator( HydrusSerialisable.SerialisableBaseNamed ):
return self._initial_search_text
def GetSafeSummary( self ):
return 'Nested downloader "' + self._name + '" - ' + ', '.join( ( name for ( gug_key, name ) in self._gug_keys_and_names ) )
def IsFunctional( self ):
for gug_key_and_name in self._gug_keys_and_names:
@ -1989,6 +2236,14 @@ class NestedGalleryURLGenerator( HydrusSerialisable.SerialisableBaseNamed ):
self._gug_keys_and_names = good_gug_keys_and_names
def SetGUGKeyAndName( self, gug_key_and_name ):
( gug_key, name ) = gug_key_and_name
self._gallery_url_generator_key = gug_key
self._name = name
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_NESTED_GALLERY_URL_GENERATOR ] = NestedGalleryURLGenerator
class URLMatch( HydrusSerialisable.SerialisableBaseNamed ):
@ -2400,6 +2655,11 @@ class URLMatch( HydrusSerialisable.SerialisableBaseNamed ):
return r.geturl()
def GetSafeSummary( self ):
return 'URL Class "' + self._name + '" - ' + ConvertURLIntoDomain( self.GetExampleURL() )
def GetURLType( self ):
return self._url_type
@ -2474,11 +2734,21 @@ class URLMatch( HydrusSerialisable.SerialisableBaseNamed ):
return is_a_direct_file_page or is_a_single_file_post_page
def RegenMatchKey( self ):
def RegenerateMatchKey( self ):
self._url_match_key = HydrusData.GenerateKey()
def SetExampleURL( self, example_url ):
self._example_url = example_url
def SetMatchKey( self, match_key ):
self._url_match_key = match_key
def ShouldAssociateWithFiles( self ):
return self._should_be_associated_with_files

View File

@ -741,6 +741,16 @@ class NetworkJob( object ):
def OverrideToken( self ):
with self._lock:
self._gallery_token_consumed = True
self._wake_time = 0
def SetError( self, e, error ):
with self._lock:

View File

@ -1937,6 +1937,15 @@ class PageParser( HydrusSerialisable.SerialisableBaseNamed ):
return self._parser_key
def GetSafeSummary( self ):
domains = list( { ClientNetworkingDomain.ConvertURLIntoDomain( url ) for url in self._example_urls } )
domains.sort()
return 'Parser "' + self._name + '" - ' + ', '.join( domains )
def GetStringConverter( self ):
return self._string_converter
@ -2084,6 +2093,21 @@ class PageParser( HydrusSerialisable.SerialisableBaseNamed ):
self._parser_key = HydrusData.GenerateKey()
def SetExampleURLs( self, example_urls ):
self._example_urls = list( example_urls )
def SetExampleParsingContext( self, example_parsing_context ):
self._example_parsing_context = example_parsing_context
def SetParserKey( self, parser_key ):
self._parser_key = parser_key
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_PAGE_PARSER ] = PageParser
class ParseNodeContentLink( HydrusSerialisable.SerialisableBase ):

View File

@ -3,6 +3,7 @@ import ClientImageHandling
import ClientImporting
import ClientParsing
import ClientPaths
import collections
import cv2
import HydrusConstants as HC
import HydrusData
@ -205,7 +206,16 @@ def GetPayloadTypeString( payload_obj ):
elif isinstance( payload_obj, HydrusSerialisable.SerialisableList ):
return 'A list of ' + HydrusData.ToHumanInt( len( payload_obj ) ) + ' ' + GetPayloadTypeString( payload_obj[0] )
type_string_counts = collections.Counter()
for o in payload_obj:
type_string_counts[ GetPayloadTypeString( o ) ] += 1
type_string = ', '.join( ( HydrusData.ToHumanInt( count ) + ' ' + s for ( s, count ) in type_string_counts.items() ) )
return 'A list of ' + type_string
elif isinstance( payload_obj, HydrusSerialisable.SerialisableBase ):

View File

@ -67,41 +67,126 @@ def RenderTag( tag, render_for_user ):
return namespace + connector + subtag
def SortTagsList( tags, sort_type ):
def SortTags( sort_by, tags_list, tags_to_count = None ):
if sort_type in ( CC.SORT_BY_LEXICOGRAPHIC_DESC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
def lexicographic_key( tag ):
reverse = True
( namespace, subtag ) = HydrusTags.SplitTag( tag )
else:
comparable_subtag = HydrusTags.ConvertTagToSortable( subtag )
reverse = False
# 'cat' < 'character:rei'
# 'page:3' < 'page:20'
# '1' < 'series:eva'
# note that 'test' < ( 1, '' ) but u'test' > ( 1, '' ) wew
if namespace == '':
return ( comparable_subtag, comparable_subtag )
else:
return ( namespace, comparable_subtag )
if sort_type in ( CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
def incidence_key( tag ):
def key( tag ):
if tags_to_count is None:
# '{' is above 'z' in ascii, so this works for most situations
return 1
( namespace, subtag ) = HydrusTags.SplitTag( tag )
else:
if namespace == '':
return tags_to_count[ tag ]
def namespace_key( tag ):
( namespace, subtag ) = HydrusTags.SplitTag( tag )
if namespace == '':
namespace = '{' # '{' is above 'z' in ascii, so this works for most situations
return namespace
def namespace_lexicographic_key( tag ):
# '{' is above 'z' in ascii, so this works for most situations
( namespace, subtag ) = HydrusTags.SplitTag( tag )
if namespace == '':
return ( '{', HydrusTags.ConvertTagToSortable( subtag ) )
else:
return ( namespace, HydrusTags.ConvertTagToSortable( subtag ) )
if sort_by in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
# let's establish a-z here for equal incidence values later
if sort_by in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC ):
tags_list.sort( key = lexicographic_key, reverse = True )
reverse = False
elif sort_by in ( CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
tags_list.sort( key = lexicographic_key )
reverse = True
tags_list.sort( key = incidence_key, reverse = reverse )
if sort_by in ( CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
# python list sort is stable, so lets now sort again
if sort_by == CC.SORT_BY_INCIDENCE_NAMESPACE_ASC:
return ( '{', subtag )
reverse = True
else:
elif sort_by == CC.SORT_BY_INCIDENCE_NAMESPACE_DESC:
return ( namespace, subtag )
reverse = False
tags_list.sort( key = namespace_key, reverse = reverse )
else:
key = None
if sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_DESC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
reverse = True
elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC ):
reverse = False
if sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
key = namespace_lexicographic_key
elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_DESC ):
key = lexicographic_key
tags_list.sort( key = key, reverse = reverse )
tags.sort( key = key, reverse = reverse )
class TagFilter( HydrusSerialisable.SerialisableBase ):

View File

@ -49,7 +49,7 @@ options = {}
# Misc
NETWORK_VERSION = 18
SOFTWARE_VERSION = 322
SOFTWARE_VERSION = 323
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )

View File

@ -51,6 +51,9 @@ class HydrusController( object ):
self._call_to_threads = []
self._long_running_call_to_threads = []
self._thread_pool_busy_status_text = ''
self._thread_pool_busy_status_text_new_check_time = 0
self._call_to_thread_lock = threading.Lock()
self._timestamps = collections.defaultdict( lambda: 0 )
@ -372,6 +375,38 @@ class HydrusController( object ):
return self._managers[ name ]
def GetThreadPoolBusyStatus( self ):
if HydrusData.TimeHasPassed( self._thread_pool_busy_status_text_new_check_time ):
with self._call_to_thread_lock:
num_threads = sum( ( 1 for t in self._call_to_threads if t.CurrentlyWorking() ) )
if num_threads < 4:
self._thread_pool_busy_status_text = ''
elif num_threads < 10:
self._thread_pool_busy_status_text = 'working'
elif num_threads < 20:
self._thread_pool_busy_status_text = 'busy'
else:
self._thread_pool_busy_status_text = 'very busy!'
self._thread_pool_busy_status_text_new_check_time = HydrusData.GetNow() + 10
return self._thread_pool_busy_status_text
def GetThreadsSnapshot( self ):
threads = []

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.6 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.0 KiB

BIN
static/lain.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB