changelog
-
+
version 294
+ - fixed video scan +
- fixed up some import folder logic--they now run 'look for new files' checks separate from 'import anything still in the queue', so they can now catch up on outstanding files more easily +
- the ten minute file-processing break is reverted--import folders now just 'save' every ten minutes, to forestall lost work on a crash +
- import folders now have an explicit 'check regularly' checkbox to control whether you want it to check regularly or only when you tell it to. the paused status now means 'do not ever work' +
- import folders now make a 'working' popup, like subscriptions do, that shows new file discovery progress and found file import progress. it has a cancel button that will stop the current job and can be hidden in the import folder's options. existing import folders will default to ON for this, so you'll likely see one of these right after you update +
- import folders can now publish their files to a new page! this effectively cuts out the button step. furthermore, if a page already exists with the import folder's name (e.g. a previous page the import folder created is still around), the import folder will publish the new files to _that_ page, updating it +
- file popup buttons with the same name will now 'merge'! multiple work cycles of a subscription or import folder will now just update the first button rather than spamming several +
- running import folders will respond faster to a client shutdown event +
- wrote some better controller-level thread management, including surplus thread deletion--idle CPU should be reduced on import-busy clients +
- wrote some code to deal with sub-second times a bit better in certain places +
- finished off a new scheduled job queue that collapses an old multiple-thread-when-idle system down into just one +
- many things are moved to the new job scheduling system--all the old calllater calls, and the popup message timers as well +
- completely removed the old wx timers the autocomplete input was using, as they were just too much of a hassle. any crashing these were causing is now gone--it all works on the simpler scheduled job queue now +
- the autocomplete dropdowns are better at judging when to engage their timers and so now use far less idle CPU +
- fixed a crash that could occur sometime after starting a duplicate filter maintenance task from the dupe filter page +
- fixed several unlikely but plausible crashes on the admin-side of petition processing +
- wrote a bunch of help for the url classes and new parsing system--it isn't finished yet, but then neither is the new system! +
- if the client fails to initialise the db, it will now try to present the error in a bit of screenshottable-gui before the program quits +
- improved thread watcher error handling when the given url is unwatchable +
- lots of timer related cleanup and tiny fixes +
- mix fixes +
version 293
- fixed the issue with options dialog not opening again after a save--I apologise for the inconvenience diff --git a/help/downloader_completion.html b/help/downloader_completion.html new file mode 100644 index 00000000..4e0fd9f1 --- /dev/null +++ b/help/downloader_completion.html @@ -0,0 +1,14 @@ + + +
Gallery Downloader
+ - This takes a string like 'blue_eyes' to produce a series of thumbnail gallery pages URLs that can be parsed for image page URLs which can ultimately be parsed for file URLs and metadata like tags. Boorus fall into this category. +
Thread Watcher
+ - This takes a URL that it will check repeatedly, parsing it for new URLs that it then queues up to be downloaded. It typically stops checking after the 'file velocity' (such as '1 new file per day') drops below a certain level. +
Single Page Downloader
+ - This takes a URL one-time and parses it for more URLs. This is a miscellaneous system for certain simple gallery types. The 'page of images' downloader is one of these. +
- Present some user interface named 'Safebooru Downloader' to the user that will convert their input of 'blue_eyes' into https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0. +
- Recognise https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0 as a Safebooru Gallery URL. +
- Convert the HTML of a Safebooru Gallery URL into a list URLs like https://safebooru.org/index.php?page=post&s=view&id=2437965 and possibly a 'next page' URL that points to another page of thumbnails. +
- Recognise https://safebooru.org/index.php?page=post&s=view&id=2437965 as a Safebooru Post URL. +
- Convert the HTML of a Safebooru Post URL into a file URL like https://safebooru.org//images/2329/b6e8c263d691d1c39a2eeba5e00709849d8f864d.jpg and some tags like: 1girl, bangs, black gloves, blonde hair, blue eyes, braid, closed mouth, day, fingerless gloves, fingernails, gloves, grass, hair ornament, hairclip, hands clasped, creator:hankuri, interlocked fingers, long hair, long sleeves, outdoors, own hands together, parted bangs, pointy ears, character:princess zelda, smile, solo, series:the legend of zelda, underbust. +
- Downloader: faces the user and converts text input into a series of Gallery URLs. +
- URL Class: identifies URLs and informs the client how to deal with them. +
- Parser: converts data from URLs into hydrus-understandable metadata. +
- Formulae: Take parsable data, search it in some manner, and return 0 to n strings. +
- Content Parser: Take parsable data, apply a formula to it to get some strings, and apply a single metadata 'type' and perhaps some additional modifiers. +
- Page Parser: Take parsable data, apply content parsers to it, and return all the metadata. +
html
+ - This takes HTML or a sample of HTML--and any regular sort of XML should also work, it is not at all strict--searches for nodes with certain tag names and/or attributes, and then returns those nodes' particular attribute value, string content, or html beneath. +
- The search occurs in steps: +
- (image of a decent formula with several steps) +
- Each step will be applied in turn, starting at the root node and searching beneath the nodes found in the previous step. +
- For instance, if you have this html: +
<html> + <body> + <div class="media_taglist"> + <span class="generaltag"><a href="(search page)">blonde hair</a> (3456)</span> + <span class="generaltag"><a href="(search page)">blue eyes</a> (4567)</span> + <span class="generaltag"><a href="(search page)">bodysuit</a> (5678)</span> + <span class="charactertag"><a href="(search page)">samus aran</a> (2345)</span> + <span class="artisttag"><a href="(search page)">splashbrush</a> (123)</span> + </div> + <div class="content"> + <span class="media">(a whole bunch of content that doesn't have tags in)</span> + </div> + </body> +</html>
+ - To find the artist, "splashbrush", you would want to: +
- get every <div> tag with attributes class=media_taglist +
- and then get every <span> tag with attributes class=artisttag +
- and then get the string content of those tags +
- This will return a single string, "splashbrush". Changing the "artisttag" to "charactertag" or "generaltag" would give you "samus aran" and "blonde hair","blue eyes","bodysuit" respectively. +
- You might be tempted to just go straight for the <span> with class=artisttag, but many sites use the same class to render a sidebar of favourite/popular tags or some other sponsored content, so it is best to make sure you narrow down to the larger <div> container so you don't get anything you don't want. +
- When you add or edit one of these rules, you get this: +
- (image of rule edit panel) +
- Note that you can select to get only the 1st or xth instance of a found tag if you like, which can be useful in situations like this: +
<span class="generaltag"> + <a href="(add tag)">+</a> + <a href="(remove tag)">-</a> + <a href="(search page)">blonde hair</a> (3456) +</span>
+ - Without any more attributes, there isn't a good way to distinguish the <a> with "blonde hair" from the other two--so just set 'get the 3rd <a> tag' and you are good. +
- Once you have narrowed down the right nodes you want, you can decide what to return. So, given a node of: +
<a href="(URL A)" class="thumb">Forest Glade</a>
+ - Returning the 'href' attribute would return the string "(URL A)", returning the string content would give "Forest Glade", and returning the full html would give "<a href="(URL A)" class="thumb">Forest Glade</a>". This last choice is useful in complicated situations where you want a second, separated layer of parsing, which we will get to later. +
json
+ - This takes some JSON and does a similar style of search: +
- (image of edit formula panel) +
- It is a bit simpler than HTML--if the current node is a list (called an 'Array' in JSON), you can fetch every item or the xth item, and if it is a dictionary (called an 'Object' in JSON), you can fetch a particular string entry. Since you can't jump down several layers with attribute lookups or tag names, you have to go down every layer one at a time. In any case, if you have something like this: +
+ - Then searching for "posts"->1st list item->"sub" will give you "Nobody like kino here.". +
- Then searching for "posts"->all list items->"tim" will give you the three file hashes (since the third post has no file attached, the parser skips over it without complaint). +
- Searching for "posts"->1st list item->"com" will give you the OP's comment, ~AS RAW UNPARSED HTML~. +
- The default is to fetch the final nodes' 'data content', which means coercing simple variables into strings. If the current node is a list or dict, no string is returned. +
- But if you like, you can return the json beneath the current node (which, like HTML, includes the current node). This again will come in useful later. +
compound
+ - If you want to create a string from multiple parsed strings--for instance by appending the 'tim' and the 'ext' in our json example together--you can use a Compound formula. This fetches multiple lists of strings and tries to place them into a single string using \1 \2 \3 regex substitution syntax: +
- (image of the edit panel--use the thread watcher one with complicated gubbins) +
- This is where the magic happens, sometimes, so keep it in mind if you need to do something cleverer than the data you have seems to provide. +
context variable
+ - desc +
- ui walkthrough +
- misc +
File URL
+ -
+
This returns the full, raw media file with no HTML wrapper. They typically end in a filename like http://safebooru.org//images/2333/cab1516a7eecf13c462615120ecf781116265f17.jpg, but sometimes they have a more complicated fetch command ending like 'file.php?id=123456' or '/post/content/123456'.
+These URLs are remembered for the file in the 'known urls' list, so if the client happens to encounter the same URL in future, it can determine whether it can skip the download because the file is already in the database or has previously been deleted.
+It is not important that File URLs are matched. The client does not need to match a File URL in order to download it (and will typically assume unmatched URLs are File URLs), but you might want to particularly specify them if you discover File URLs are being confused for Post URLs or something.
+
+ Post URL
+ -
+
This typically contains one File URL and some metadata like tags. They sometimes present multiple sizes (like 'sample' vs 'full size') of the file or even different formats (like 'ugoira' vs 'webm'). The Post URL for the file above, http://safebooru.org/index.php?page=post&s=view&id=2429668 has this 'sample' presentation. Finding the best File URL in these cases can be tricky!
+This URL is also saved to 'known urls' and can be skipped if it has previously been downloaded. It will also appear in the media viewer as a clickable link. Since the user may want to load these in their browser, it is important that they stay intact and valid.
+
+ Gallery URL
+ - This presents a list of Post URLs or File URLs. They often also present a 'next page' URL and sometimes say when the files it links were posted, which can be used to calculate how fast files are being posted. It could be a page like http://safebooru.org/index.php?page=post&s=list&tags=yorha_no._2_type_b&pid=0 or an API URL like http://safebooru.org/index.php?page=dapi&s=post&tags=yorha_no._2_type_b&q=index&pid=0. +
Watchable URL
+ - This is the same as a Gallery URL but represents an ephemeral page that receives new files much faster than a gallery but will soon 'die' and be deleted. For our purposes, this typically means imageboard threads. +
- Scheme: "http" or "https" +
- Location/Domain: "safebooru.org" or "i.4cdn.org" or "cdn002.somebooru.net" +
- Path Components: "index.php" or "tesla/res/7518.json" or "pictures/user/daruak/page/2" or "art/Commission-animation-Elsa-and-Anna-541820782" +
- Query Parameters: "page=post&s=list&tags=yorha_no._2_type_b&pid=40" or "page=post&s=view&id=2429668" +
Scheme
+ -
+
TBIB supports http and https, so I have set the 'preferred' scheme to https. Any 'http' TBIB URL a user inputs will be automatically converted to https.
+
+ Location/Domain
+ -
+
For File URLs, the domain is always "tbib.org".
+The 'allow' and 'keep' subdomains checkboxes let you determine if a URL with "rule34.booru.org" will match a URL Class with "booru.org" domain and if that subdomain should be remembered going forward. In the booru.org case, the subdomain is very important, and removing it breaks the URL, but in cases where a site farms out File URLs to CDN servers on subdomains (like randomly serving a mirror of "https://muhbooru.org/file/123456" on "https://srv2.muhbooru.org/file/123456") and removing the subdomain still gives a valid URL, you do not wish to keep the subdomain. As it is, TBIB does not use subdomains, so these options do not matter in this case.
+I am not totally happy with how allow and keep subdomains work, so I may replace this in future.
+
+ Path Components
+ -
+
TBIB just uses a single "index.php" on the root directory, so the path is not complicated. Were it longer (like "gallery/cgi/index.php", we would add more ("gallery" and "cgi"), and since the path of a URL has a strict order, we would need to arrange the items in the listbox there so they were sorted correctly.
+
+ Query Parameters
+ -
+
TBIB's index.php takes many query parameters to render different page types. Note that the Post URL uses "s=view", while TBIB Gallery URLs use "s=list". In any case, for a Post URL, "id", "page", and "s" are necessary and sufficient.
+
+ - URL A: https://8ch.net/tv/res/1002432.html +
- URL B: https://8ch.net/tv/res +
- URL C: https://8ch.net/tv/res/1002432 +
- URL D: https://8ch.net/tv/res/1002432.json +
- URL Class that looks for "(characters)/res/(numbers).html" for the path +
- URL A: https://boards.4chan.org/m/thread/16086187 +
- URL B: https://boards.4chan.org/m/thread/16086187/ssg-super-sentai-general-651 +
- URL Class that looks for "(characters)/thread/(numbers)" for the path +
- URL A: https://www.pixiv.net/member_illust.php?mode=medium&illust_id=66476204 +
- URL B: https://www.pixiv.net/member_illust.php?mode=medium&illust_id=66476204&lang=jp +
- URL C: https://www.pixiv.net/member_illust.php?mode=medium +
- URL Class that looks for "illust_id=(numbers)" in the query +
- URL A: https://somebooru.com/post/123456 +
- URL B: https://somebooru.com/post/123456/manga_subpage/2 +
- URL Class A that looks for "post/(number)" for the path +
- URL Class B that looks for "post/(number)/manga_subpage/(number)" for the path +
- https://www.hentai-foundry.com/pictures/user/Sparrow/scraps/page/3 +
- https://www.hentai-foundry.com/pictures/user/Sparrow/scraps/page/1 +
- https://www.hentai-foundry.com/pictures/user/Sparrow/scraps (note that this gives the same result as the /page/1 URL) +
- running a client or server in wine
- running a client or server from source
making a downloader (under construction)
+ - introduction +
- url classes +
- parsers +
- downloaders +
- putting it all together +
- sharing downloaders +
- login manager +
misc
- privacy diff --git a/help/json_thread_example.png b/help/json_thread_example.png new file mode 100644 index 00000000..511c76da Binary files /dev/null and b/help/json_thread_example.png differ diff --git a/include/ClientCaches.py b/include/ClientCaches.py index 68a35038..9d21cd8a 100644 --- a/include/ClientCaches.py +++ b/include/ClientCaches.py @@ -11,6 +11,7 @@ import HydrusFileHandling import HydrusPaths import HydrusSerialisable import HydrusSessions +import HydrusThreading import itertools import json import os @@ -2017,7 +2018,7 @@ class ThumbnailCache( object ): last_paused = HydrusData.GetNowPrecise() - while not HG.view_shutdown: + while not HydrusThreading.IsThreadShuttingDown(): with self._lock: diff --git a/include/ClientController.py b/include/ClientController.py index 4334501c..4007082c 100755 --- a/include/ClientController.py +++ b/include/ClientController.py @@ -167,6 +167,28 @@ class Controller( HydrusController.HydrusController ): raise HydrusExceptions.ShutdownException() + def CallLaterWXSafe( self, window, delay, func, *args, **kwargs ): + + call = HydrusData.Call( func, *args, **kwargs ) + + job = ClientThreading.WXAwareJob( self, self._job_scheduler, window, call, initial_delay = delay ) + + self._job_scheduler.AddJob( job ) + + return job + + + def CallRepeatingWXSafe( self, window, period, delay, func, *args, **kwargs ): + + call = HydrusData.Call( func, *args, **kwargs ) + + job = ClientThreading.WXAwareRepeatingJob( self, self._job_scheduler, window, call, period, initial_delay = delay ) + + self._job_scheduler.AddJob( job ) + + return job + + def CheckAlreadyRunning( self ): while HydrusData.IsAlreadyRunning( self.db_dir, 'client' ): @@ -370,14 +392,19 @@ class Controller( HydrusController.HydrusController ): with ClientGUIDialogs.DialogYesNo( self._splash, text, title = 'Maintenance is due' ) as dlg_yn: - call_later = ClientThreading.CallLater( dlg_yn, 15, dlg_yn.EndModal, wx.ID_NO ) + job = self.CallLaterWXSafe( dlg_yn, 15, dlg_yn.EndModal, wx.ID_NO ) - if dlg_yn.ShowModal() == wx.ID_YES: + try: - HG.do_idle_shutdown_work = True + if dlg_yn.ShowModal() == wx.ID_YES: + + HG.do_idle_shutdown_work = True + + + finally: + + job.Cancel() - - call_later.Stop() else: diff --git a/include/ClientDB.py b/include/ClientDB.py index 6abc22c6..ae6d8e2f 100755 --- a/include/ClientDB.py +++ b/include/ClientDB.py @@ -3190,6 +3190,17 @@ class DB( HydrusDB.HydrusDB ): + def _DisplayCatastrophicError( self, text ): + + message = 'The db encountered a serious error! This is going to be written to the log as well, but here it is for a screenshot:' + message += os.linesep * 2 + message += text + + HydrusData.DebugPrint( message ) + + wx.SafeShowMessage( message ) + + def _ExportToTagArchive( self, path, service_key, hash_type, hashes = None ): # This could nicely take a whitelist or a blacklist for namespace filtering diff --git a/include/ClientDaemons.py b/include/ClientDaemons.py index 444b4aac..ac927804 100644 --- a/include/ClientDaemons.py +++ b/include/ClientDaemons.py @@ -45,7 +45,7 @@ def DAEMONCheckImportFolders( controller ): import_folder = controller.Read( 'serialisable_named', HydrusSerialisable.SERIALISABLE_TYPE_IMPORT_FOLDER, name ) - if controller.options[ 'pause_import_folders_sync' ] or HG.view_shutdown: + if controller.options[ 'pause_import_folders_sync' ] or HydrusThreading.IsThreadShuttingDown(): break diff --git a/include/ClientGUI.py b/include/ClientGUI.py index adcef6ca..7dc0e711 100755 --- a/include/ClientGUI.py +++ b/include/ClientGUI.py @@ -127,6 +127,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ): self._controller.sub( self, 'NotifyNewServices', 'notify_new_services_gui' ) self._controller.sub( self, 'NotifyNewSessions', 'notify_new_sessions' ) self._controller.sub( self, 'NotifyNewUndo', 'notify_new_undo' ) + self._controller.sub( self, 'PresentImportedFilesToPage', 'imported_files_to_page' ) self._controller.sub( self, 'RenamePage', 'rename_page' ) self._controller.sub( self, 'SetDBLockedStatus', 'db_locked_status' ) self._controller.sub( self, 'SetMediaFocus', 'set_media_focus' ) @@ -141,7 +142,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ): self.SetSizer( vbox ) - ClientGUITopLevelWindows.SetTLWSizeAndPosition( self, self._frame_key ) + ClientGUITopLevelWindows.SetInitialTLWSizeAndPosition( self, self._frame_key ) self.Show( True ) @@ -555,7 +556,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ): self._notebook.SaveGUISession( 'last session' ) # session save causes a db read in the menu refresh, so let's put this off just a bit - ClientThreading.CallLater( self, 1.5, self._controller.Write, 'backup', path ) + self._controller.CallLater( 1.5, self._controller.Write, 'backup', path ) @@ -760,8 +761,9 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ): self._controller.pub( 'message', job_key ) - ClientThreading.CallLater( self, 2, job_key.SetVariable, 'popup_text_2', 'Pulsing subjob' ) - ClientThreading.CallLater( self, 2, job_key.SetVariable, 'popup_gauge_2', ( 0, None ) ) + self._controller.CallLater( 2.0, job_key.SetVariable, 'popup_text_2', 'Pulsing subjob' ) + + self._controller.CallLater( 2.0, job_key.SetVariable, 'popup_gauge_2', ( 0, None ) ) # @@ -773,7 +775,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ): for i in range( 1, 4 ): - ClientThreading.CallLater( self, 0.5 * i, HydrusData.ShowText, 'This is a delayed popup message -- ' + str( i ) ) + self._controller.CallLater( 0.5 * i, HydrusData.ShowText, 'This is a delayed popup message -- ' + str( i ) ) @@ -1636,7 +1638,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ): ClientGUIMenus.AppendMenu( debug, report_modes, 'report modes' ) ClientGUIMenus.AppendMenuItem( self, debug, 'make some popups', 'Throw some varied popups at the message manager, just to check it is working.', self._DebugMakeSomePopups ) - ClientGUIMenus.AppendMenuItem( self, debug, 'make a popup in five seconds', 'Throw a delayed popup at the message manager, giving you time to minimise or otherwise alter the client before it arrives.', ClientThreading.CallLater, self, 5, HydrusData.ShowText, 'This is a delayed popup message.' ) + ClientGUIMenus.AppendMenuItem( self, debug, 'make a popup in five seconds', 'Throw a delayed popup at the message manager, giving you time to minimise or otherwise alter the client before it arrives.', self._controller.CallLater, 5, HydrusData.ShowText, 'This is a delayed popup message.' ) ClientGUIMenus.AppendMenuItem( self, debug, 'force a gui layout now', 'Tell the gui to relayout--useful to test some gui bootup layout issues.', self.Layout ) ClientGUIMenus.AppendMenuItem( self, debug, 'flush log', 'Command the log to write any buffered contents to hard drive.', HydrusData.DebugPrint, 'Flushing log' ) ClientGUIMenus.AppendMenuItem( self, debug, 'print garbage', 'Print some information about the python garbage to the log.', self._DebugPrintGarbage ) @@ -1863,7 +1865,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ): last_session_save_period_minutes = self._controller.new_options.GetInteger( 'last_session_save_period_minutes' ) - ClientThreading.CallLater( self, last_session_save_period_minutes * 60, self.SaveLastSession ) + self._controller.CallLaterWXSafe( self, last_session_save_period_minutes * 60, self.SaveLastSession ) def _ManageAccountTypes( self, service_key ): @@ -3174,7 +3176,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p if self.IsIconized(): - ClientThreading.CallLater( self, 10, self.AddModalMessage, job_key ) + self._controller.CallLaterWXSafe( self, 10, self.AddModalMessage, job_key ) else: @@ -3460,16 +3462,19 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p with ClientGUIDialogs.DialogYesNo( self, text ) as dlg: - timer = ClientThreading.CallLater( self, 15, dlg.EndModal, wx.ID_YES ) + job = self._controller.CallLaterWXSafe( dlg, 15, dlg.EndModal, wx.ID_YES ) - if dlg.ShowModal() == wx.ID_NO: + try: - timer.Stop() + if dlg.ShowModal() == wx.ID_NO: + + return False + - return False + finally: + + job.Cancel() - - timer.Stop() @@ -3533,7 +3538,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p else: - ClientThreading.CallLater( self, 2, self.Destroy ) + self._controller.CallLaterWXSafe( self, 2, self.Destroy ) self._controller.CreateSplash() @@ -3825,6 +3830,11 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p return False + def PresentImportedFilesToPage( self, hashes, page_name ): + + dest_page = self._notebook.PresentImportedFilesToPage( hashes, page_name ) + + def RefreshMenu( self ): if not self: @@ -3836,7 +3846,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p if db_going_to_hang_if_we_hit_it: - ClientThreading.CallLater( self, 2.5, self.RefreshMenu ) + self._controller.CallLaterWXSafe( self, 0.5, self.RefreshMenu ) return @@ -3934,7 +3944,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p last_session_save_period_minutes = self._controller.new_options.GetInteger( 'last_session_save_period_minutes' ) - ClientThreading.CallLater( self, last_session_save_period_minutes * 60, self.SaveLastSession ) + self._controller.CallLaterWXSafe( self, last_session_save_period_minutes * 60, self.SaveLastSession ) def SetMediaFocus( self ): diff --git a/include/ClientGUIACDropdown.py b/include/ClientGUIACDropdown.py index 49e2f64b..f9b77f38 100644 --- a/include/ClientGUIACDropdown.py +++ b/include/ClientGUIACDropdown.py @@ -7,6 +7,7 @@ import ClientGUIMenus import ClientSearch import collections import HydrusConstants as HC +import HydrusData import HydrusExceptions import HydrusGlobals as HG import HydrusTags @@ -120,19 +121,13 @@ class AutoCompleteDropdown( wx.Panel ): self._initial_matches_fetched = False - self._move_hide_timer = None + self._move_hide_job = None if self._float_mode: self.Bind( wx.EVT_MOVE, self.EventMove ) self.Bind( wx.EVT_SIZE, self.EventMove ) - self.Bind( wx.EVT_TIMER, self.TIMEREventDropdownHide, id = ID_TIMER_DROPDOWN_HIDE ) - - self._move_hide_timer = wx.Timer( self, id = ID_TIMER_DROPDOWN_HIDE ) - - self._move_hide_timer.Start( 1, wx.TIMER_ONE_SHOT ) - tlp.Bind( wx.EVT_MOVE, self.EventMove ) parent = self @@ -155,13 +150,11 @@ class AutoCompleteDropdown( wx.Panel ): - self.Bind( wx.EVT_TIMER, self.TIMEREventLag, id = ID_TIMER_AC_LAG ) - HG.client_controller.sub( self, '_UpdateBackgroundColour', 'notify_new_colourset' ) - self._lag_timer = wx.Timer( self, id = ID_TIMER_AC_LAG ) + self._refresh_list_job = None - wx.CallAfter( self._UpdateList ) + self._ScheduleListRefresh( 0.0 ) def _BroadcastChoices( self, predicates ): @@ -176,6 +169,14 @@ class AutoCompleteDropdown( wx.Panel ): self._BroadcastChoices( { text } ) + def _CancelScheduledListRefresh( self ): + + if self._refresh_list_job is not None: + + self._refresh_list_job.Cancel() + + + def _GenerateMatches( self ): raise NotImplementedError() @@ -196,6 +197,27 @@ class AutoCompleteDropdown( wx.Panel ): raise NotImplementedError() + def _ScheduleListRefresh( self, delay ): + + if self._refresh_list_job is not None and delay == 0.0: + + self._refresh_list_job.MoveNextWorkTimeToNow() + + else: + + self._CancelScheduledListRefresh() + + self._refresh_list_job = HG.client_controller.CallLaterWXSafe( self, delay, self._UpdateList ) + + + + def _SetListDirty( self ): + + self._cache_text = None + + self._ScheduleListRefresh( 0.0 ) + + def _ShouldShow( self ): tlp_active = self.GetTopLevelParent().IsActive() or self._dropdown_window.IsActive() @@ -309,18 +331,38 @@ class AutoCompleteDropdown( wx.Panel ): self._BroadcastChoices( predicates ) - def CleanBeforeDestroy( self ): + def DropdownHideShow( self ): - if self._move_hide_timer is not None: + try: - self._move_hide_timer.Stop() - self._move_hide_timer = None + should_show = self._ShouldShow() - - if self._lag_timer is not None: + if should_show: + + self._ShowDropdown() + + if self._move_hide_job is not None: + + self._move_hide_job.Cancel() + + self._move_hide_job = None + + + else: + + self._HideDropdown() + - self._lag_timer.Stop() - self._lag_timer = None + except: + + if self._move_hide_job is not None: + + self._move_hide_job.Cancel() + + self._move_hide_job = None + + + raise @@ -338,9 +380,7 @@ class AutoCompleteDropdown( wx.Panel ): elif key == wx.WXK_SPACE and event.RawControlDown(): # this is control, not command on os x, for which command+space does some os stuff - self._UpdateList() - - self._lag_timer.Stop() + self._ScheduleListRefresh( 0.0 ) elif self._intercept_key_events: @@ -397,9 +437,9 @@ class AutoCompleteDropdown( wx.Panel ): def EventKillFocus( self, event ): - if self._move_hide_timer: + if self._float_mode: - self._move_hide_timer.Start( 1, wx.TIMER_ONE_SHOT ) + self.DropdownHideShow() event.Skip() @@ -453,11 +493,19 @@ class AutoCompleteDropdown( wx.Panel ): def EventMove( self, event ): - self._HideDropdown() - - if self._move_hide_timer: + if self._float_mode: - self._move_hide_timer.Start( 250, wx.TIMER_ONE_SHOT ) + self._HideDropdown() + + if self._ShouldShow(): + + if self._move_hide_job is None: + + self._move_hide_job = HG.client_controller.CallRepeatingWXSafe( self._dropdown_window, 0.25, 0.0, self.DropdownHideShow ) + + + self._move_hide_job.Delay( 0.25 ) + event.Skip() @@ -465,9 +513,9 @@ class AutoCompleteDropdown( wx.Panel ): def EventSetFocus( self, event ): - if self._move_hide_timer: + if self._float_mode: - self._move_hide_timer.Start( 1, wx.TIMER_ONE_SHOT ) + self.DropdownHideShow() event.Skip() @@ -479,7 +527,7 @@ class AutoCompleteDropdown( wx.Panel ): if num_chars == 0: - self._UpdateList() + self._ScheduleListRefresh( 0.0 ) elif HC.options[ 'fetch_ac_results_automatically' ]: @@ -487,70 +535,21 @@ class AutoCompleteDropdown( wx.Panel ): self._next_updatelist_is_probably_fast = self._next_updatelist_is_probably_fast and num_chars > len( self._last_search_text ) - if self._next_updatelist_is_probably_fast: self._UpdateList() + if self._next_updatelist_is_probably_fast: + + self._ScheduleListRefresh( 0.0 ) + elif num_chars < char_limit: - self._lag_timer.Start( long_wait, wx.TIMER_ONE_SHOT ) + self._ScheduleListRefresh( long_wait / 1000.0 ) else: - self._lag_timer.Start( short_wait, wx.TIMER_ONE_SHOT ) + self._ScheduleListRefresh( short_wait / 1000.0 ) - def RefreshList( self ): - - self._cache_text = None - - self._UpdateList() - - - def TIMEREventDropdownHide( self, event ): - - try: - - should_show = self._ShouldShow() - - if should_show: - - self._ShowDropdown() - - else: - - self._HideDropdown() - - - if self._move_hide_timer: - - self._move_hide_timer.Start( 250, wx.TIMER_ONE_SHOT ) - - - except: - - if self._move_hide_timer: - - self._move_hide_timer.Stop() - - - raise - - - - def TIMEREventLag( self, event ): - - try: - - self._UpdateList() - - except: - - self._lag_timer.Stop() - - raise - - - class AutoCompleteDropdownTags( AutoCompleteDropdown ): def __init__( self, parent, file_service_key, tag_service_key ): @@ -588,7 +587,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ): self._file_repo_button.SetLabelText( name ) - wx.CallAfter( self.RefreshList ) + self._SetListDirty() def _ChangeTagService( self, tag_service_key ): @@ -610,11 +609,13 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ): self._cache_text = None - wx.CallAfter( self.RefreshList ) + self._SetListDirty() def _UpdateList( self ): + self._refresh_list_job = None + self._last_search_text = self._text_ctrl.GetValue() matches = self._GenerateMatches() @@ -629,7 +630,9 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ): if num_chars == 0: - self._lag_timer.Start( 5 * 60 * 1000, wx.TIMER_ONE_SHOT ) + # refresh system preds after five mins + + self._ScheduleListRefresh( 300 ) @@ -1060,7 +1063,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ): self._file_search_context.SetIncludeCurrentTags( value ) - wx.CallAfter( self.RefreshList ) + self._SetListDirty() HG.client_controller.pub( 'refresh_query', self._page_key ) @@ -1072,7 +1075,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ): self._file_search_context.SetIncludePendingTags( value ) - wx.CallAfter( self.RefreshList ) + self._SetListDirty() HG.client_controller.pub( 'refresh_query', self._page_key ) diff --git a/include/ClientGUICanvas.py b/include/ClientGUICanvas.py index 6359a89e..d73c219b 100755 --- a/include/ClientGUICanvas.py +++ b/include/ClientGUICanvas.py @@ -1109,7 +1109,7 @@ class CanvasFrame( ClientGUITopLevelWindows.FrameThatResizes ): self.SetSizer( vbox ) - ClientGUITopLevelWindows.SetTLWSizeAndPosition( self, self._frame_key ) + ClientGUITopLevelWindows.SetInitialTLWSizeAndPosition( self, self._frame_key ) self.Show( True ) @@ -2757,7 +2757,7 @@ class CanvasWithHovers( CanvasWithDetails ): # - self._timer_cursor_hide = ClientThreading.WXAwareTimer( self, self.TIMERCursorHide ) + self._timer_cursor_hide_job = None self.Bind( wx.EVT_MOTION, self.EventDrag ) @@ -2863,7 +2863,7 @@ class CanvasWithHovers( CanvasWithDetails ): self.SetCursor( wx.Cursor( wx.CURSOR_ARROW ) ) - self._timer_cursor_hide.CallLater( 0.8 ) + self._PutOffCursorHide() else: @@ -2871,29 +2871,30 @@ class CanvasWithHovers( CanvasWithDetails ): - def TIMERCursorHide( self ): + def _PutOffCursorHide( self ): - try: + if self._timer_cursor_hide_job is not None: - if not CC.CAN_HIDE_MOUSE: - - return - + self._timer_cursor_hide_job.Cancel() - if HG.client_controller.MenuIsOpen(): - - self._timer_cursor_hide.CallLater( 0.8 ) - - else: - - self.SetCursor( wx.Cursor( wx.CURSOR_BLANK ) ) - + + self._timer_cursor_hide_job = HG.client_controller.CallLaterWXSafe( self, 0.8, self._HideCursor ) + + + def _HideCursor( self ): + + if not CC.CAN_HIDE_MOUSE: - except: + return - self._timer_cursor_hide.Stop() + + if HG.client_controller.MenuIsOpen(): - raise + self._PutOffCursorHide() + + else: + + self.SetCursor( wx.Cursor( wx.CURSOR_BLANK ) ) @@ -3624,20 +3625,17 @@ class CanvasFilterDuplicates( CanvasWithHovers ): # ugly, but it will do for now - if self: + if len( self._media_list ) < 2: - if len( self._media_list ) < 2: - - self._ShowNewPair() - - else: - - self._SetDirty() - + self._ShowNewPair() + + else: + + self._SetDirty() - ClientThreading.CallLater( self, 0.1, catch_up ) + HG.client_controller.CallLaterWXSafe( self, 0.1, catch_up ) def SetMedia( self, media ): @@ -3829,7 +3827,7 @@ class CanvasMediaList( ClientMedia.ListeningMediaList, CanvasWithHovers ): if not image_cache.HasImageRenderer( hash ): - ClientThreading.CallLater( self, delay, image_cache.GetImageRenderer, media ) + HG.client_controller.CallLaterWXSafe( self, delay, image_cache.GetImageRenderer, media ) @@ -3925,7 +3923,7 @@ class CanvasMediaList( ClientMedia.ListeningMediaList, CanvasWithHovers ): def KeepCursorAlive( self ): - self._timer_cursor_hide.CallLater( 0.8 ) + self._PutOffCursorHide() def ProcessContentUpdates( self, service_keys_to_content_updates ): @@ -4451,7 +4449,7 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ): CanvasMediaListNavigable.__init__( self, parent, page_key, media_results ) - self._timer_slideshow = ClientThreading.WXAwareTimer( self, self.TIMERSlideshow ) + self._timer_slideshow_job = None self._timer_slideshow_interval = 0 self.Bind( wx.EVT_LEFT_DCLICK, self.EventClose ) @@ -4482,19 +4480,19 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ): def _PausePlaySlideshow( self ): - if self._timer_slideshow.IsRunning(): + if self._timer_slideshow_job is not None: - self._timer_slideshow.Stop() + self._StopSlideshow() elif self._timer_slideshow_interval > 0: - self._timer_slideshow.CallLater( self._timer_slideshow_interval, repeating = True ) + self._StartSlideshow( self._timer_slideshow_interval ) def _StartSlideshow( self, interval = None ): - self._timer_slideshow.Stop() + self._StopSlideshow() if interval is None: @@ -4518,7 +4516,43 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ): self._timer_slideshow_interval = interval - self._timer_slideshow.CallLater( self._timer_slideshow_interval, repeating = True ) + self._timer_slideshow_job = HG.client_controller.CallLaterWXSafe( self, self._timer_slideshow_interval, self.DoSlideshow ) + + + + def _StopSlideshow( self ): + + if self._timer_slideshow_job is not None: + + self._timer_slideshow_job.Cancel() + + self._timer_slideshow_job = None + + + + def DoSlideshow( self ): + + try: + + if self._current_media is not None and self._timer_slideshow_job is not None: + + if self._media_container.ReadyToSlideshow() and not HG.client_controller.MenuIsOpen(): + + self._ShowNext() + + self._timer_slideshow_job = HG.client_controller.CallLaterWXSafe( self, self._timer_slideshow_interval, self.DoSlideshow ) + + else: + + self._timer_slideshow_job = HG.client_controller.CallLaterWXSafe( self, 0.5, self.DoSlideshow ) + + + + except: + + self._timer_slideshow_job = None + + raise @@ -4720,7 +4754,7 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ): ClientGUIMenus.AppendMenu( menu, slideshow, 'start slideshow' ) - if self._timer_slideshow.IsRunning(): + if self._timer_slideshow_job is not None: ClientGUIMenus.AppendMenuItem( self, menu, 'stop slideshow', 'Stop the current slideshow.', self._PausePlaySlideshow ) @@ -4742,30 +4776,6 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ): event.Skip() - def TIMERSlideshow( self ): - - try: - - if self._current_media is not None: - - if self._media_container.ReadyToSlideshow() and not HG.client_controller.MenuIsOpen(): - - self._ShowNext() - - else: - - self._timer_slideshow.Delay( 0.5 ) - - - - except: - - self._timer_slideshow.Stop() - - raise - - - class MediaContainer( wx.Window ): def __init__( self, parent ): diff --git a/include/ClientGUIDialogs.py b/include/ClientGUIDialogs.py index f65c9f62..127f9fcd 100755 --- a/include/ClientGUIDialogs.py +++ b/include/ClientGUIDialogs.py @@ -664,7 +664,7 @@ class DialogInputLocalBooruShare( Dialog ): else: - time_left = max( 0, timeout - HydrusData.GetNow() ) + time_left = HydrusData.GetTimeDeltaUntilTime( timeout ) if time_left < 60 * 60 * 12: time_value = 60 elif time_left < 60 * 60 * 24 * 7: time_value = 60 * 60 diff --git a/include/ClientGUIDialogsManage.py b/include/ClientGUIDialogsManage.py index 5b69a599..591050e2 100644 --- a/include/ClientGUIDialogsManage.py +++ b/include/ClientGUIDialogsManage.py @@ -2411,7 +2411,7 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ): self._import_folder = import_folder - ( name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, open_popup, paused, check_now ) = self._import_folder.ToTuple() + ( name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, check_regularly, paused, check_now, show_working_popup, publish_files_to_popup_button, publish_files_to_page ) = self._import_folder.ToTuple() self._panel = wx.ScrolledWindow( self ) @@ -2421,7 +2421,7 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ): self._path = wx.DirPickerCtrl( self._folder_box, style = wx.DIRP_USE_TEXTCTRL ) - self._open_popup = wx.CheckBox( self._folder_box ) + self._check_regularly = wx.CheckBox( self._folder_box ) self._period = ClientGUITime.TimeDeltaButton( self._folder_box, min = 3 * 60, days = True, hours = True, minutes = True ) @@ -2429,6 +2429,10 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ): self._check_now = wx.CheckBox( self._folder_box ) + self._show_working_popup = wx.CheckBox( self._folder_box ) + self._publish_files_to_popup_button = wx.CheckBox( self._folder_box ) + self._publish_files_to_page = wx.CheckBox( self._folder_box ) + self._seed_cache_button = ClientGUISeedCache.SeedCacheButton( self._folder_box, HG.client_controller, self._import_folder.GetSeedCache, seed_cache_set_callable = self._import_folder.SetSeedCache ) # @@ -2496,11 +2500,16 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ): self._name.SetValue( name ) self._path.SetPath( path ) - self._open_popup.SetValue( open_popup ) + + self._check_regularly.SetValue( check_regularly ) self._period.SetValue( period ) self._paused.SetValue( paused ) + self._show_working_popup.SetValue( show_working_popup ) + self._publish_files_to_popup_button.SetValue( publish_files_to_popup_button ) + self._publish_files_to_page.SetValue( publish_files_to_page ) + self._mimes.SetValue( mimes ) self._action_successful.SelectClientData( actions[ CC.STATUS_SUCCESSFUL ] ) @@ -2539,10 +2548,13 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ): rows.append( ( 'name: ', self._name ) ) rows.append( ( 'folder path: ', self._path ) ) + rows.append( ( 'currently paused (if set, will not ever do any work): ', self._paused ) ) + rows.append( ( 'check regularly?: ', self._check_regularly ) ) rows.append( ( 'check period: ', self._period ) ) - rows.append( ( 'periodic checks currently paused: ', self._paused ) ) rows.append( ( 'check on manage dialog ok: ', self._check_now ) ) - rows.append( ( 'open a popup if new files imported: ', self._open_popup ) ) + rows.append( ( 'show a popup while working: ', self._show_working_popup ) ) + rows.append( ( 'if new files imported, publish them to a popup button: ', self._publish_files_to_popup_button ) ) + rows.append( ( 'if new files imported, publish them to a page: ', self._publish_files_to_page ) ) rows.append( ( 'review currently cached import paths: ', self._seed_cache_button ) ) gridbox = ClientGUICommon.WrapInGrid( self._folder_box, rows ) @@ -2621,6 +2633,10 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ): self._CheckLocations() + self._check_regularly.Bind( wx.EVT_CHECKBOX, self.EventCheckRegularly ) + + self._UpdateCheckRegularly() + wx.CallAfter( self._ok.SetFocus ) @@ -2753,6 +2769,23 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ): + def _UpdateCheckRegularly( self ): + + if self._check_regularly.GetValue(): + + self._period.Enable() + + else: + + self._period.Disable() + + + + def EventCheckRegularly( self, event ): + + self._UpdateCheckRegularly() + + def EventCheckLocations( self, event ): self._CheckLocations() @@ -2888,15 +2921,19 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ): period = self._period.GetValue() - open_popup = self._open_popup.GetValue() + check_regularly = self._check_regularly.GetValue() paused = self._paused.GetValue() check_now = self._check_now.GetValue() + show_working_popup = self._show_working_popup.GetValue() + publish_files_to_popup_button = self._publish_files_to_popup_button.GetValue() + publish_files_to_page = self._publish_files_to_page.GetValue() + tag_service_keys_to_filename_tagging_options = dict( self._filename_tagging_options.GetData() ) - self._import_folder.SetTuple( name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, open_popup, paused, check_now ) + self._import_folder.SetTuple( name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, check_regularly, paused, check_now, show_working_popup, publish_files_to_popup_button, publish_files_to_page ) return self._import_folder @@ -3006,7 +3043,7 @@ class DialogManagePixivAccount( ClientGUIDialogs.Dialog ): self._status.SetLabelText( 'OK!' ) - ClientThreading.CallLater( self, 5, self._status.SetLabel, '' ) + HG.client_controller.CallLaterWXSafe( self._status, 5, self._status.SetLabel, '' ) else: diff --git a/include/ClientGUIManagement.py b/include/ClientGUIManagement.py index 45da29c3..32572285 100755 --- a/include/ClientGUIManagement.py +++ b/include/ClientGUIManagement.py @@ -1207,7 +1207,7 @@ class ManagementPanelDuplicateFilter( ManagementPanel ): time.sleep( 0.25 ) - self._RefreshAndUpdateStatus() + wx.CallAfter( self._RefreshAndUpdateStatus ) def EventSearchDistanceChanged( self, event ): @@ -2597,36 +2597,6 @@ class ManagementPanelPetitions( ManagementPanel ): self._controller.sub( self, 'RefreshQuery', 'refresh_query' ) - def _BreakApprovedContentsIntoChunks( self, approved_contents ): - - chunks_of_approved_contents = [] - chunk_of_approved_contents = [] - weight = 0 - - for content in approved_contents: - - chunk_of_approved_contents.append( content ) - - weight += content.GetVirtualWeight() - - if weight > 50: - - chunks_of_approved_contents.append( chunk_of_approved_contents ) - - chunk_of_approved_contents = [] - - weight = 0 - - - - if len( chunk_of_approved_contents ) > 0: - - chunks_of_approved_contents.append( chunk_of_approved_contents ) - - - return chunks_of_approved_contents - - def _CheckAll( self ): for i in range( self._contents.GetCount() ): @@ -2754,25 +2724,47 @@ class ManagementPanelPetitions( ManagementPanel ): def _FetchNumPetitions( self ): - def do_it(): + def do_it( service ): + + def wx_draw( n_p_i ): + + if not self: + + return + + + self._num_petition_info = n_p_i + + self._DrawNumPetitions() + + + def wx_reset(): + + if not self: + + return + + + self._refresh_num_petitions_button.SetLabelText( 'refresh counts' ) + try: - response = self._service.Request( HC.GET, 'num_petitions' ) + response = service.Request( HC.GET, 'num_petitions' ) - self._num_petition_info = response[ 'num_petitions' ] + num_petition_info = response[ 'num_petitions' ] - wx.CallAfter( self._DrawNumPetitions ) + wx.CallAfter( wx_draw, num_petition_info ) finally: - self._refresh_num_petitions_button.SetLabelText( 'refresh counts' ) + wx.CallAfter( wx_reset ) self._refresh_num_petitions_button.SetLabelText( u'Fetching\u2026' ) - self._controller.CallToThread( do_it ) + self._controller.CallToThread( do_it, self._service ) def _FetchPetition( self, content_type, status ): @@ -2802,11 +2794,11 @@ class ManagementPanelPetitions( ManagementPanel ): button.SetLabelText( 'fetch ' + HC.content_status_string_lookup[ status ] + ' ' + HC.content_type_string_lookup[ content_type ] + ' petition' ) - def do_it(): + def do_it( service ): try: - response = self._service.Request( HC.GET, 'petition', { 'content_type' : content_type, 'status' : status } ) + response = service.Request( HC.GET, 'petition', { 'content_type' : content_type, 'status' : status } ) wx.CallAfter( wx_setpet, response[ 'petition' ] ) @@ -2826,7 +2818,7 @@ class ManagementPanelPetitions( ManagementPanel ): button.Disable() button.SetLabelText( u'Fetching\u2026' ) - self._controller.CallToThread( do_it ) + self._controller.CallToThread( do_it, self._service ) def _FlipSelected( self ): @@ -2876,7 +2868,37 @@ class ManagementPanelPetitions( ManagementPanel ): def EventProcess( self, event ): - def do_it( approved_contents, denied_contents, petition ): + def break_approved_contents_into_chunks( approved_contents ): + + chunks_of_approved_contents = [] + chunk_of_approved_contents = [] + weight = 0 + + for content in approved_contents: + + chunk_of_approved_contents.append( content ) + + weight += content.GetVirtualWeight() + + if weight > 50: + + chunks_of_approved_contents.append( chunk_of_approved_contents ) + + chunk_of_approved_contents = [] + + weight = 0 + + + + if len( chunk_of_approved_contents ) > 0: + + chunks_of_approved_contents.append( chunk_of_approved_contents ) + + + return chunks_of_approved_contents + + + def do_it( controller, service, petition_service_key, approved_contents, denied_contents, petition ): try: @@ -2901,7 +2923,7 @@ class ManagementPanelPetitions( ManagementPanel ): job_key = None - chunks_of_approved_contents = self._BreakApprovedContentsIntoChunks( approved_contents ) + chunks_of_approved_contents = break_approved_contents_into_chunks( approved_contents ) for chunk_of_approved_contents in chunks_of_approved_contents: @@ -2919,9 +2941,9 @@ class ManagementPanelPetitions( ManagementPanel ): ( update, content_updates ) = petition.GetApproval( chunk_of_approved_contents ) - self._service.Request( HC.POST, 'update', { 'client_to_server_update' : update } ) + service.Request( HC.POST, 'update', { 'client_to_server_update' : update } ) - self._controller.WriteSynchronous( 'content_updates', { self._petition_service_key : content_updates } ) + controller.WriteSynchronous( 'content_updates', { petition_service_key : content_updates } ) num_done += len( chunk_of_approved_contents ) @@ -2940,7 +2962,7 @@ class ManagementPanelPetitions( ManagementPanel ): update = petition.GetDenial( denied_contents ) - self._service.Request( HC.POST, 'update', { 'client_to_server_update' : update } ) + service.Request( HC.POST, 'update', { 'client_to_server_update' : update } ) finally: @@ -2950,7 +2972,17 @@ class ManagementPanelPetitions( ManagementPanel ): job_key.Delete() - wx.CallAfter( self._FetchNumPetitions ) + def wx_fetch(): + + if not self: + + return + + + self._FetchNumPetitions() + + + wx.CallAfter( wx_fetch ) @@ -2971,7 +3003,7 @@ class ManagementPanelPetitions( ManagementPanel ): - HG.client_controller.CallToThread( do_it, approved_contents, denied_contents, self._current_petition ) + HG.client_controller.CallToThread( do_it, self._controller, self._service, self._petition_service_key, approved_contents, denied_contents, self._current_petition ) self._current_petition = None @@ -3133,11 +3165,6 @@ class ManagementPanelQuery( ManagementPanel ): self._query_job_key.Cancel() - if self._search_enabled: - - self._searchbox.CleanBeforeDestroy() - - def GetPredicates( self ): diff --git a/include/ClientGUIPages.py b/include/ClientGUIPages.py index 3bbd138d..28f14cdc 100755 --- a/include/ClientGUIPages.py +++ b/include/ClientGUIPages.py @@ -455,7 +455,7 @@ class Page( wx.SplitterWindow ): self._media_panel.Hide() # If this is a CallAfter, OS X segfaults on refresh jej - ClientThreading.CallLater( self, 0.5, self._media_panel.Destroy ) + self._controller.CallLaterWXSafe( self._media_panel, 0.5, self._media_panel.Destroy ) self._media_panel = new_panel @@ -1059,6 +1059,29 @@ class PagesNotebook( wx.Notebook ): + return None + + + def _GetPageFromName( self, page_name ): + + for page in self._GetPages(): + + if page.GetDisplayName() == page_name: + + return page + + + if isinstance( page, PagesNotebook ): + + result = page._GetPageFromName( page_name ) + + if result is not None: + + return result + + + + return None @@ -1934,7 +1957,7 @@ class PagesNotebook( wx.Notebook ): - def NewPage( self, management_controller, initial_hashes = None, forced_insertion_index = None, on_deepest_notebook = False ): + def NewPage( self, management_controller, initial_hashes = None, forced_insertion_index = None, on_deepest_notebook = False, select_page = True ): current_page = self.GetCurrentPage() @@ -2013,7 +2036,7 @@ class PagesNotebook( wx.Notebook ): # in some unusual circumstances, this gets out of whack insertion_index = min( insertion_index, self.GetPageCount() ) - self.InsertPage( insertion_index, page, page_name, select = True ) + self.InsertPage( insertion_index, page, page_name, select = select_page ) self._controller.pub( 'refresh_page_name', page.GetPageKey() ) self._controller.pub( 'notify_new_pages' ) @@ -2079,7 +2102,7 @@ class PagesNotebook( wx.Notebook ): return self.NewPage( management_controller, on_deepest_notebook = on_deepest_notebook ) - def NewPageQuery( self, file_service_key, initial_hashes = None, initial_predicates = None, page_name = None, on_deepest_notebook = False, do_sort = False ): + def NewPageQuery( self, file_service_key, initial_hashes = None, initial_predicates = None, page_name = None, on_deepest_notebook = False, do_sort = False, select_page = True ): if initial_hashes is None: @@ -2111,7 +2134,7 @@ class PagesNotebook( wx.Notebook ): management_controller = ClientGUIManagement.CreateManagementControllerQuery( page_name, file_service_key, file_search_context, search_enabled ) - page = self.NewPage( management_controller, initial_hashes = initial_hashes, on_deepest_notebook = on_deepest_notebook ) + page = self.NewPage( management_controller, initial_hashes = initial_hashes, on_deepest_notebook = on_deepest_notebook, select_page = select_page ) if do_sort: @@ -2338,6 +2361,28 @@ class PagesNotebook( wx.Notebook ): + def PresentImportedFilesToPage( self, hashes, page_name ): + + page = self._GetPageFromName( page_name ) + + if page is None: + + page = self.NewPageQuery( CC.LOCAL_FILE_SERVICE_KEY, initial_hashes = hashes, page_name = page_name, on_deepest_notebook = True, select_page = False ) + + else: + + unsorted_media_results = self._controller.Read( 'media_results', hashes ) + + hashes_to_media_results = { media_result.GetHash() : media_result for media_result in unsorted_media_results } + + sorted_media_results = [ hashes_to_media_results[ hash ] for hash in hashes ] + + page.GetMediaPanel().AddMediaResults( page.GetPageKey(), sorted_media_results ) + + + return page + + def RefreshAllPages( self ): for page in self._GetPages(): diff --git a/include/ClientGUIParsing.py b/include/ClientGUIParsing.py index 2fad78d0..4d3b0c3d 100644 --- a/include/ClientGUIParsing.py +++ b/include/ClientGUIParsing.py @@ -218,7 +218,7 @@ class EditCompoundFormulaPanel( ClientGUIScrolledPanels.EditPanel ): The substitution phrase works like in regexes. If you have two formulae that produce _filename_ and _ext_, you could set this: -\1.\2''' +\\1.\\2''' info_st = wx.StaticText( info_panel, label = message ) @@ -903,9 +903,9 @@ The html's branches will be searched recursively by each tag rule in turn and th So, to find the 'src' of the first
-
+
+
+
+
\ No newline at end of file
diff --git a/help/downloader_login.html b/help/downloader_login.html
new file mode 100644
index 00000000..6fe54b28
--- /dev/null
+++ b/help/downloader_login.html
@@ -0,0 +1,14 @@
+
+
+ This system and help is all under construction! Even when it is done, it will be for advanced users who understand HTML or JSON. Beware!
+this system
+The first versions of hydrus's downloaders were all hardcoded and static--I wrote everything into the program itself and nothing was user-creatable or -fixable. After the maintenance burden of the entire messy system proved too large for me to keep up with and a semi-editable booru system proved successful, I decided to significantly overhaul the whole thing. The new system allows user creation and sharing of every component. It is designed to be very simple to the front-end user--they will typically handle a couple of png files and then select a new downloader from a list--but very flexible (and hence pretty complicated) on the back-end. These help pages describe the different compontents with the intention of making an HTML- or JSON- fluent user able to create and share a full new downloader on their own.
+As always, this is all under active development. Your feedback on the system would be appreciated, and if something is confusing or you discover something in here that is out of date, please let me know.
+what is a downloader?
+In hydrus, a downloader is one of:
+-
+
The system currently supports HTML and JSON parsing.
+what does a downloader do?
+As an example, in order for hydrus to convert our 'blue_eyes' query into a bunch of files with tags, it needs to:
+-
+
So we have three components:
+-
+
Thread watchers and single page downloaders do not need the 'Downloader' component, as the input in this case is a URL. You drop an imageboard thread URL on the client and it automatically recognises what it is, launches a thread watcher page for it, and finds the correct parser for the output.
+ +
+
+
+
+
\ No newline at end of file
diff --git a/help/downloader_sharing.html b/help/downloader_sharing.html
new file mode 100644
index 00000000..ddcd607e
--- /dev/null
+++ b/help/downloader_sharing.html
@@ -0,0 +1,14 @@
+
+
+ This system still needs work. The user interface remains a hellscape, so I won't put in screenshots for now.
+parsers
+In hydrus, a parser is an object that takes a single block of HTML or JSON data (as returned by a URL) and returns many kinds of hydrus-level metadata.
+Parsers are flexible and potentially complicated. You might like to open network->manage parsers and explore the UI as you read this page. Check out how the examples in the client work, and if you want to write a new one, see if there is something already in there that is similar--it is usually easier to duplicate an existing parser and then alter it than to create a new one from scratch every time.
+There are three main components in the parsing system:
+-
+
Formulae do the grunt work of parsing and string conversion, content parsers turn the strings into something richer, and page parsers are the containers. +
formulae
+A formula takes some data and returns some strings. The different kinds are:
+-
+
-
+
talk about string match and string converter
+how to test
+It is a great idea to check the html or json you are trying to parse with your browser. Most web browsers have great developer tools that let you walk through the different nodes in a pretty way. The JSON image above is one of the views Firefox provides if you simply enter a JSON URL.
+content parser
+different types and what they mean
+hash needs conversion to bytes
+vetos
+page parser
+pre-parsing conversion example for tumblr
+example urls are helpful
+mention vetos again
+subsidiary page parsers and what that is for
+page example
+do a danbooru example with sample image stuff
+gallery example
+something with an API?
+thread example
+subsidiary page parsers in the example
+source time and subject->comment fallback fun
+The context variable bit to fetch the right board for the file url
+ +
+
+
+
\ No newline at end of file
diff --git a/help/edit_string_converter_panel.png b/help/edit_string_converter_panel.png
new file mode 100644
index 00000000..c5511221
Binary files /dev/null and b/help/edit_string_converter_panel.png differ
diff --git a/help/edit_string_match_panel.png b/help/edit_string_match_panel.png
new file mode 100644
index 00000000..0444c426
Binary files /dev/null and b/help/edit_string_match_panel.png differ
diff --git a/help/index.html b/help/index.html
index 21ebea90..598a7984 100755
--- a/help/index.html
+++ b/help/index.html
@@ -48,6 +48,16 @@
<---- Back to the introduction
+url classes
+The fundamental connective part of the downloader system is the 'URL Class'. This object identifies and normalises URLs and links them to other components. When the client handles or presents a URL, it consults the respective URL Class on what to do.
+the types of url
+For hydrus, an URL is useful if it is one of:
+-
+
the components of a url
+For our purposes, a URL string has four parts:
+-
+
So, let's look at the 'edit url class' panel, which is found under network->manage url classes:
+
A TBIB File Page like https://tbib.org/index.php?page=post&s=view&id=6391256 is a Post URL. Let's go over the four components again:
+-
+
This URL Class will be assigned to any URL that matches the location, path, and query. Missing components in the URL will invalidate the match but additonal components will not!
+For instance:
+-
+
Only URL A will match
+And:
+-
+
Both URL A and B will match
+And:
+-
+
Both URL A and B will match, URL C will not
+If multiple URL Classes match a URL, the client will try to assign the most 'complicated' one, with the most path components and then query parameters.
+Given two example URLs and URL Classes:
+-
+
URL A will match URL Class A but not URL Class B and so will receive A.
+URL B will match both and will receive URL Class B as it is more complicated.
+This situation is not common, but I expect it to be an issue with Pixiv, where some Post URLs link to a subset of manga pages that have their own gallery system, wew.
+string matches
+As you edit these components, you will be presented with the Edit String Match Panel:
+
This lets you set the type of string that will be valid for that component. If a given path or query component does not match the rules given here, the URL will not match the URL Class. Most of the time you will probably want to set 'fixed characters' of something like "post" or "index.php", but if the component you are editing is more complicated and could have a range of different valid values, you can specify just numbers or letters or even a regex pattern. If you try to do something complicated, experiment with the 'example string' entry to make sure you have it set how you think.
+Don't go overboard with this stuff, though--most sites do not have super-fine distinctions between their different URL types, and hydrus users will not be dropping user account or logout pages or whatever on the client, so you can be fairly liberal with the rules.
+normalising urls
+Different URLs can give the same page. The http and https versions of a URL are typically the same, and "http://site.com/index.php?s=post&id=123456" results in the same content as "http://site.com/index.php?id=123456&s=post", and "https://e621.net/post/show/1421754/abstract_background-animal_humanoid-blush-brown_ey" is the same as "https://e621.net/post/show/1421754".
+Since we are in the business of storing and comparing URLs, we want to 'normalise' them to a single comparable beautiful value. You see a preview of this normalisation on the edit panel.
+Gallery and Watchable URLs are not compared, so a normalise call for them only switches their http/https to the preferred value, but File and Post URLs will cut out any surplus path or query components and will alphabetise the query arguments as well.
+Since File and Post URLs will remove anything surplus, be careful that you not leave out anything important in your rules. Make sure what you have is both necessary (nothing can be removed and still keep it valid) and sufficient (no more needs to be added to make it valid). It is a good idea to try pasting the 'normalised' version of the example URL into your browser, just to check it still works.
+gallery rules do not need to be sufficient
+Advanced--feel free to skip for now
+For Gallery URLs, however, it can sometimes be useful to specify just a set of necessary rules. This saves your time and covers a broader set of URLs like these:
+-
+
Rather than making two rules--one with the additional "/page/(number)" and one without--you can just make one for "pictures/user/(characters)/scraps", which will match all three examples above.
+While hydrus downloaders tend to generate valid first page URLs with something like "/page/1" or "pid=0" or "index=0", the sites themselves tend to link a 'bare' URL to a user browsing with a mouse. If you demand the 'page' or 'index' part in your Gallery URL Classes, a user who finds a nice gallery and tries to drop the first page's URL, as the site presented it, onto the client will only get a 'Couldn't find a URL Class for that!' error.
+But if there isn't a nice way to create a single non-ambiguous class, just make multiple.
+api urls
+If you know that a URL has an API backend, you can tell the client to use that API URL when it fetches data. The API URL needs its own URL Class.
+To define the relationship, click the "String Converter" button, which gives you this:
+
You may have seen this panel elsewhere. It lets you convert a string to another over a number of transformation steps. The steps can be as simple as adding or removing some characters or applying a full regex substitution. For API URLs, you are mostly looking to isolate some unique identifying data ("m/thread/16086187" in this case) and then substituting that into the new API path. It is worth testing this with several different examples!
+When the client links regular URLs to API URLs like this, it will still associate the human-pretty regular URL when it needs to display to the user and record 'known urls' and so on. The API is just a quick lookup when it actually fetches and parses the respective data.
+ +-
+