changelog
-
+
version 435
+ - misc: +
- a new macOS build that should run on Big Sur is now ready, it should be attached to this release. it is built on github automatically, and is thanks to hard work from Suika and ReAnzu. I am attaching my old release as well, just in case I messed up somewhere on my end. if you are a macOS user, please try the new App! it will not work on very old macOS like 10.12, but if this works out today for the majority of macOS users, I will be moving to just putting this new build out going forward. I'll add some polish like the readme.rtf and harmonise the filename etc.. too. I'd love to cut the filesize down, but this may not be possible (it is apparently some modern macOS thing where it bundles old and new versions of libraries in the same App so you basically get it twice) +
- the bottom-right corner of the regular media viewer canvas now also shows media zoom +
- the StringSorter object now has a simple 'reverse' sort type +
- the infamous multi-column list 'last column' width calculations are improved: first, dialogs with multi-column lists should no longer judder back and forth a single character's width as you expand the parent window. also, the last column saved size (which is used in dialog relaunch width initialisation) is now snapped to rounded 5-character intervals, which should mitigate various 'fuzzy' reasons for some dialogs to remember a larger or smaller size and grow or shrink one or more characters' width on the next launch +
- the _help->debug->gui actions_ menu has a new entry to reset all multi-column list saved widths back to default +
- the 'edit OR predicate' panel when you shift+double-click an OR predicate now expands horizontally and vertically with the window +
- the 'edit search predicates' list in the 'edit favourite search' panel now expands vertically with the window +
- the client now detects some invalid tag mapping states on tag upload--when a mapping is both current & pending or when it is both deleted and petitioned. these pair-states are mutually exclusive, normally impossible to get to, but one user who nonetheless ended up in this situation encountered an infinite uploading loop to a tag repository (since the tag was already current/deleted, the pending/petitioned status was not clearing correctly on upload commit). now, the upload will be abandoned and an info message put up with the fix +
- added a new maintenance routine to _database->check and repair_ that fixes logically inconsistent mappings. it has a popup dialog when it works and forces a pending count refresh and shows a summary afterwards +
- the routine that counts up total current or pending mappings on a service when the cached number has been reset is now massively faster (from a 30-60s down to less than a second in my dev tests). it now sums the tag autocomplete cache, rather than counting raw tables +
- fixed the BUGFIX option in 'connections' that allows you to disable ssl verification. this will also be extended at a later date to be domain-specific +
- . +
- new server stuff: +
- a new permission is added to hydrus service accounts--'manage options'. any account with 'manage account types' will get this by default on update +
- any account on a repository with 'manage options' permission will now see 'change update period' in the admin services menu! it launches a time delta control with the current update period and will send the new one up to the server. the client will resync account, options, and metadata immediately, and the server will generate any now-due updates immediately, so you should be able to watch changes occur in 'review services' and the server terminal live. other users will catch up to the new time when they next hit an update. various hardcoded check periods (like how often due updates are checked for and delay-buffered clientside and serverside) are shrunk significantly. the whole system should react to changes better +
- the minimum settable update time is now 10 minutes (the default value remains 100,000 seconds), but I recommend you try larger, say an hour minimum, at least to start. the network generally works more efficiently with higher numbers, and be warned, if you are adding 144 updates a day, there may be bloat problems after a year +
- let me know how this goes, whether you are running a server on a LAN or just a regular user running on one who gets a new update time! +
- the new 'full metadata resync' routine now triggers an immediate metadata update sync and wakes the daemon involved, so it should now happen as you watch +
- fixed the new pause/play buttons on review services to use neutral pause/play icons, not the downloader pause/play +
- brushed up metadata sync status string on review services +
- cleaned misc server and network code +
- cleaned up some old clientside service code +
- the client api now supports wildcard and namespace tags in the file search call +
- client api version is now 16 +
- added https://ififfy.github.io/flipflip/#/ , a slideshow engine that now supports hydrus as a source, to the client api page +
version 434
- network updates: @@ -27,14 +56,14 @@
- fixed an issue with the new file sort asc/desc button where a transition from 'random' to another sort type using a favourite search would always reset the sort order to the top value
- my asynchronous job object now has a default errback to catch errors more gracefully by default and with special handling in future. clicking an async button in a dialog will now show you the error there and then, rather than just the hidden error popup on the main window
- added convenience links to the latest build on github to the help menu and html help -
- fixed another place in local file importing where a file that did not pass file import options checks would set 'skipped' status. in now sets 'ignored' like everything else +
- fixed another place in local file importing where a file that did not pass file import options checks would set 'skipped' status. it now sets 'ignored' like everything else
- fixed a bug when an 'undelete' call is sent to the media viewer when no media is set (usually during startup/shutdown)
- I disabled progress gauge 'pulsing' across the program. the way this was first implemented applied too often--I will bring it back to only apply when a job is both indeterminate and currently working
- my custom button class can now launch its own yes/no confirmation dialogs on click
- removed a subtag regen routine in the 425->426 update step that was bugging out due to bitrot--it now makes a popup message on boot asking for the routine to be run manually
- fixed a typo bug in the 'subscription snapshot' debug command
- misc ancient python 2-to-3 code cleanup -
- updaned cloudscraper to 1.2.58 +
- updated cloudscraper to 1.2.58
- .
- admin stuff:
- . diff --git a/help/client_api.html b/help/client_api.html index 3a9cb4be..20d175e5 100644 --- a/help/client_api.html +++ b/help/client_api.html @@ -18,7 +18,8 @@
- https://gitgud.io/prkc/hydrus-companion - Hydrus Companion, a Chrome/Firefox extension for hydrus that allows easy download queueing as you browse and advanced login support
- https://github.com/floogulinc/hydrus-web - Hydrus Web, a web client for hydrus (allows phone browsing of hydrus) -
- https://www.animebox.es/ - Anime Boxes now supports adding your client as a Hydrus Server +
- https://www.animebox.es/ - Anime Boxes, a booru browser, now supports adding your client as a Hydrus Server +
- https://ififfy.github.io/flipflip/#/ - FlipFlip, an advanced slideshow interface, now supports hydrus as a source
- https://gitgud.io/koto/hydrus-archive-delete - Archive/Delete filter in your web browser
- https://gitgud.io/koto/hydrus-dd - DeepDanbooru neural network tagging for Hydrus
- https://gitgud.io/prkc/dolphin-hydrus-actions - Adds Hydrus right-click context menu actions to Dolphin file manager. @@ -42,7 +43,7 @@
- @@ -983,7 +984,8 @@
/get_files/search_files?system_inbox=true&tags=%5B%22blue%20eyes%22%2C%20%22blonde%20hair%22%2C%20%22%5Cu043a%5Cu0438%5Cu043d%5Cu043e%22%5D
-
Example response:
diff --git a/hydrus/client/ClientController.py b/hydrus/client/ClientController.py index 71f57eb3..ffe712b8 100644 --- a/hydrus/client/ClientController.py +++ b/hydrus/client/ClientController.py @@ -21,6 +21,7 @@ from hydrus.core import HydrusPaths from hydrus.core import HydrusSerialisable from hydrus.core import HydrusThreading from hydrus.core import HydrusVideoHandling +from hydrus.core.networking import HydrusNetwork from hydrus.core.networking import HydrusNetworking from hydrus.client import ClientAPI @@ -1184,9 +1185,10 @@ class Controller( HydrusController.HydrusController ): job = self.CallRepeating( 5.0, 3600.0, self.SynchroniseAccounts ) job.ShouldDelayOnWakeup( True ) job.WakeOnPubSub( 'notify_unknown_accounts' ) + job.WakeOnPubSub( 'notify_new_permissions' ) self._daemon_jobs[ 'synchronise_accounts' ] = job - job = self.CallRepeating( 5.0, 3600.0 * 4, self.SynchroniseRepositories ) + job = self.CallRepeating( 5.0, HydrusNetwork.UPDATE_CHECKING_PERIOD, self.SynchroniseRepositories ) job.ShouldDelayOnWakeup( True ) job.WakeOnPubSub( 'notify_restart_repo_sync' ) job.WakeOnPubSub( 'notify_new_permissions' ) diff --git a/hydrus/client/ClientParsing.py b/hydrus/client/ClientParsing.py index f241ac0b..5112442f 100644 --- a/hydrus/client/ClientParsing.py +++ b/hydrus/client/ClientParsing.py @@ -1865,6 +1865,7 @@ HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIAL CONTENT_PARSER_SORT_TYPE_NONE = 0 CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC = 1 CONTENT_PARSER_SORT_TYPE_HUMAN_SORT = 2 +CONTENT_PARSER_SORT_TYPE_REVERSE = 3 class ContentParser( HydrusSerialisable.SerialisableBase ): @@ -3872,7 +3873,8 @@ HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIAL sort_str_enum = { CONTENT_PARSER_SORT_TYPE_NONE : 'no sorting', CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC : 'strict lexicographic', - CONTENT_PARSER_SORT_TYPE_HUMAN_SORT : 'human sort' + CONTENT_PARSER_SORT_TYPE_HUMAN_SORT : 'human sort', + CONTENT_PARSER_SORT_TYPE_REVERSE : 'reverse' } class StringSorter( StringProcessingStep ): @@ -3926,49 +3928,56 @@ class StringSorter( StringProcessingStep ): texts = list( texts ) - data_convert = lambda d_s: d_s - invalid_data_convert_texts = [] - - if self._regex is not None: + if self._sort_type == CONTENT_PARSER_SORT_TYPE_REVERSE: - re_job = re.compile( self._regex ) + texts.reverse() - def d( d_s ): + else: + + data_convert = lambda d_s: d_s + invalid_data_convert_texts = [] + + if self._regex is not None: - m = re_job.search( d_s ) + re_job = re.compile( self._regex ) - if m is None: + def d( d_s ): - return '' + m = re_job.search( d_s ) - else: - - return m.group() + if m is None: + + return '' + + else: + + return m.group() + + data_convert = d + + invalid_data_convert_texts = [ text for text in texts if data_convert( text ) == '' ] + texts = [ text for text in texts if data_convert( text ) != '' ] + - data_convert = d + sort_convert = lambda s: s - invalid_data_convert_texts = [ text for text in texts if data_convert( text ) == '' ] - texts = [ text for text in texts if data_convert( text ) != '' ] + if self._sort_type == CONTENT_PARSER_SORT_TYPE_HUMAN_SORT: + + sort_convert = HydrusData.HumanTextSortKey + - - sort_convert = lambda s: s - - if self._sort_type == CONTENT_PARSER_SORT_TYPE_HUMAN_SORT: + key = lambda k_s: sort_convert( data_convert( k_s ) ) - sort_convert = HydrusData.HumanTextSortKey + reverse = not self._asc + + texts.sort( key = key, reverse = reverse ) + + invalid_data_convert_texts.sort( key = sort_convert, reverse = reverse ) + + texts.extend( invalid_data_convert_texts ) - - key = lambda k_s: sort_convert( data_convert( k_s ) ) - - reverse = not self._asc - - texts.sort( key = key, reverse = reverse ) - - invalid_data_convert_texts.sort( key = sort_convert, reverse = reverse ) - - texts.extend( invalid_data_convert_texts ) return texts diff --git a/hydrus/client/ClientServices.py b/hydrus/client/ClientServices.py index de73d7e7..908e1a5c 100644 --- a/hydrus/client/ClientServices.py +++ b/hydrus/client/ClientServices.py @@ -2124,8 +2124,11 @@ class ServiceRepository( ServiceRestricted ): with self._lock: + self._next_account_sync = 1 self._do_a_full_metadata_resync = True + self._metadata.UpdateASAP() + self._SetDirty() @@ -2158,11 +2161,18 @@ class ServiceRepository( ServiceRestricted ): - def GetUpdateInfo( self ): + def GetUpdatePeriod( self ): with self._lock: - return self._metadata.GetUpdateInfo() + if 'update_period' in self._service_options: + + return self._service_options[ 'update_period' ] + + else: + + raise HydrusExceptions.DataMissing( 'This service does not seem to have an update period! Try refreshing your account!' ) + diff --git a/hydrus/client/db/ClientDB.py b/hydrus/client/db/ClientDB.py index 60774a07..fdc567c2 100644 --- a/hydrus/client/db/ClientDB.py +++ b/hydrus/client/db/ClientDB.py @@ -7617,6 +7617,92 @@ class DB( HydrusDB.HydrusDB ): + def _FixLogicallyInconsistentMappings( self, tag_service_key = None ): + + job_key = ClientThreading.JobKey( cancellable = True ) + + total_fixed = 0 + + try: + + job_key.SetVariable( 'popup_title', 'fixing logically inconsistent mappings' ) + + self._controller.pub( 'modal_message', job_key ) + + if tag_service_key is None: + + tag_service_ids = self.modules_services.GetServiceIds( HC.REAL_TAG_SERVICES ) + + else: + + tag_service_ids = ( self.modules_services.GetServiceId( tag_service_key ), ) + + + for tag_service_id in tag_service_ids: + + if job_key.IsCancelled(): + + break + + + message = 'fixing {}'.format( tag_service_id ) + + job_key.SetVariable( 'popup_text_1', message ) + + time.sleep( 0.01 ) + + ( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = ClientDBMappingsStorage.GenerateMappingsTableNames( tag_service_id ) + + # + + both_current_and_pending_mappings = list( + HydrusData.BuildKeyToSetDict( + self._c.execute( 'SELECT tag_id, hash_id FROM {} CROSS JOIN {} USING ( tag_id, hash_id );'.format( pending_mappings_table_name, current_mappings_table_name ) ) + ).items() + ) + + total_fixed += sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in both_current_and_pending_mappings ) ) + + self._UpdateMappings( tag_service_id, pending_rescinded_mappings_ids = both_current_and_pending_mappings ) + + # + + both_deleted_and_petitioned_mappings = list( + HydrusData.BuildKeyToSetDict( + self._c.execute( 'SELECT tag_id, hash_id FROM {} CROSS JOIN {} USING ( tag_id, hash_id );'.format( petitioned_mappings_table_name, deleted_mappings_table_name ) ) + ).items() + ) + + total_fixed += sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in both_deleted_and_petitioned_mappings ) ) + + self._UpdateMappings( tag_service_id, petitioned_rescinded_mappings_ids = both_deleted_and_petitioned_mappings ) + + + finally: + + if total_fixed == 0: + + HydrusData.ShowText( 'No inconsistent mappings found!' ) + + else: + + self._c.execute( 'DELETE FROM service_info where info_type IN ( ?, ? );', ( HC.SERVICE_INFO_NUM_PENDING_MAPPINGS, HC.SERVICE_INFO_NUM_PETITIONED_MAPPINGS ) ) + + self._controller.pub( 'notify_new_pending' ) + + HydrusData.ShowText( 'Found {} bad mappings! They _should_ be deleted, and your pending counts should be updated.'.format( HydrusData.ToHumanInt( total_fixed ) ) ) + + + job_key.DeleteVariable( 'popup_text_2' ) + + job_key.SetVariable( 'popup_text_1', 'done!' ) + + job_key.Finish() + + job_key.Delete( 5 ) + + + def _GenerateDBJob( self, job_type, synchronous, action, *args, **kwargs ): return JobDatabaseClient( job_type, synchronous, action, *args, **kwargs ) @@ -11030,7 +11116,26 @@ class DB( HydrusDB.HydrusDB ): pending_dict = HydrusData.BuildKeyToListDict( self._c.execute( 'SELECT tag_id, hash_id FROM ' + pending_mappings_table_name + ' ORDER BY tag_id LIMIT 100;' ) ) - for ( tag_id, hash_ids ) in list(pending_dict.items()): + pending_mapping_ids = list( pending_dict.items() ) + + # dealing with a scary situation when (due to some bug) mappings are current and pending. they get uploaded, but the content update makes no changes, so we cycle infitely! + addable_pending_mapping_ids = self._FilterExistingUpdateMappings( service_id, pending_mapping_ids, HC.CONTENT_UPDATE_ADD ) + + pending_mapping_weight = sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in pending_mapping_ids ) ) + addable_pending_mapping_weight = sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in addable_pending_mapping_ids ) ) + + if pending_mapping_weight != addable_pending_mapping_weight: + + message = 'Hey, while going through the pending tags to upload, it seemed some were simultaneously already in the \'current\' state. This looks like a bug.' + message += os.linesep * 2 + message += 'Please run _database->check and repair->fix logically inconsistent mappings_. If everything seems good after that and you do not get this message again, you should be all fixed. If not, you may need to regenerate your mappings storage cache under the \'database\' menu. If that does not work, hydev would like to know about it!' + + HydrusData.ShowText( message ) + + raise HydrusExceptions.VetoException( 'Logically inconsistent mappings detected!' ) + + + for ( tag_id, hash_ids ) in pending_mapping_ids: tag = self.modules_tags_local_cache.GetTag( tag_id ) hashes = self.modules_hashes_local_cache.GetHashes( hash_ids ) @@ -11042,7 +11147,30 @@ class DB( HydrusDB.HydrusDB ): petitioned_dict = HydrusData.BuildKeyToListDict( [ ( ( tag_id, reason_id ), hash_id ) for ( tag_id, hash_id, reason_id ) in self._c.execute( 'SELECT tag_id, hash_id, reason_id FROM ' + petitioned_mappings_table_name + ' ORDER BY reason_id LIMIT 100;' ) ] ) - for ( ( tag_id, reason_id ), hash_ids ) in list(petitioned_dict.items()): + petitioned_mapping_ids = list( petitioned_dict.items() ) + + # dealing with a scary situation when (due to some bug) mappings are deleted and petitioned. they get uploaded, but the content update makes no changes, so we cycle infitely! + deletable_and_petitioned_mappings = self._FilterExistingUpdateMappings( + service_id, + [ ( tag_id, hash_ids ) for ( ( tag_id, reason_id ), hash_ids ) in petitioned_mapping_ids ], + HC.CONTENT_UPDATE_DELETE + ) + + petitioned_mapping_weight = sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in petitioned_mapping_ids ) ) + deletable_petitioned_mapping_weight = sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in deletable_and_petitioned_mappings ) ) + + if petitioned_mapping_weight != deletable_petitioned_mapping_weight: + + message = 'Hey, while going through the petitioned tags to upload, it seemed some were simultaneously already in the \'deleted\' state. This looks like a bug.' + message += os.linesep * 2 + message += 'Please run _database->check and repair->fix logically inconsistent mappings_. If everything seems good after that and you do not get this message again, you should be all fixed. If not, you may need to regenerate your mappings storage cache under the \'database\' menu. If that does not work, hydev would like to know about it!' + + HydrusData.ShowText( message ) + + raise HydrusExceptions.VetoException( 'Logically inconsistent mappings detected!' ) + + + for ( ( tag_id, reason_id ), hash_ids ) in petitioned_mapping_ids: tag = self.modules_tags_local_cache.GetTag( tag_id ) hashes = self.modules_hashes_local_cache.GetHashes( hash_ids ) @@ -11127,7 +11255,7 @@ class DB( HydrusDB.HydrusDB ): return media_result - petitioned = list(HydrusData.BuildKeyToListDict( self._c.execute( 'SELECT reason_id, hash_id FROM file_petitions WHERE service_id = ? ORDER BY reason_id LIMIT 100;', ( service_id, ) ) ).items()) + petitioned = list( HydrusData.BuildKeyToListDict( self._c.execute( 'SELECT reason_id, hash_id FROM file_petitions WHERE service_id = ? ORDER BY reason_id LIMIT 100;', ( service_id, ) ) ).items() ) for ( reason_id, hash_ids ) in petitioned: @@ -11640,9 +11768,27 @@ class DB( HydrusDB.HydrusDB ): result = self._c.execute( 'SELECT COUNT( * ) FROM {};'.format( tags_table_name ) ).fetchone() - elif info_type == HC.SERVICE_INFO_NUM_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + current_mappings_table_name + ';' ).fetchone() + elif info_type in ( HC.SERVICE_INFO_NUM_MAPPINGS, HC.SERVICE_INFO_NUM_PENDING_MAPPINGS ): + + ac_cache_table_name = self._CacheMappingsGetACCacheTableName( ClientTags.TAG_DISPLAY_STORAGE, self.modules_services.combined_file_service_id, service_id ) + + if info_type == HC.SERVICE_INFO_NUM_MAPPINGS: + + column_name = 'current_count' + + elif info_type == HC.SERVICE_INFO_NUM_PENDING_MAPPINGS: + + column_name = 'pending_count' + + + result = self._c.execute( 'SELECT SUM( {} ) FROM {};'.format( column_name, ac_cache_table_name ) ).fetchone() + + if result is None or result[0] is None: + + result = ( 0, ) + + elif info_type == HC.SERVICE_INFO_NUM_DELETED_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + deleted_mappings_table_name + ';' ).fetchone() - elif info_type == HC.SERVICE_INFO_NUM_PENDING_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + pending_mappings_table_name + ';' ).fetchone() elif info_type == HC.SERVICE_INFO_NUM_PETITIONED_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + petitioned_mappings_table_name + ';' ).fetchone() elif info_type == HC.SERVICE_INFO_NUM_PENDING_TAG_SIBLINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM tag_sibling_petitions WHERE service_id = ? AND status = ?;', ( service_id, HC.CONTENT_STATUS_PENDING ) ).fetchone() elif info_type == HC.SERVICE_INFO_NUM_PETITIONED_TAG_SIBLINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM tag_sibling_petitions WHERE service_id = ? AND status = ?;', ( service_id, HC.CONTENT_STATUS_PETITIONED ) ).fetchone() @@ -19298,6 +19444,7 @@ class DB( HydrusDB.HydrusDB ): elif action == 'file_maintenance_add_jobs_hashes': self._FileMaintenanceAddJobsHashes( *args, **kwargs ) elif action == 'file_maintenance_cancel_jobs': self._FileMaintenanceCancelJobs( *args, **kwargs ) elif action == 'file_maintenance_clear_jobs': self._FileMaintenanceClearJobs( *args, **kwargs ) + elif action == 'fix_logically_inconsistent_mappings': self._FixLogicallyInconsistentMappings( *args, **kwargs ) elif action == 'imageboard': self.modules_serialisable.SetYAMLDump( ClientDBSerialisable.YAML_DUMP_ID_IMAGEBOARD, *args, **kwargs ) elif action == 'ideal_client_files_locations': self._SetIdealClientFilesLocations( *args, **kwargs ) elif action == 'import_file': result = self._ImportFile( *args, **kwargs ) diff --git a/hydrus/client/gui/ClientGUI.py b/hydrus/client/gui/ClientGUI.py index a7828578..28a6df63 100644 --- a/hydrus/client/gui/ClientGUI.py +++ b/hydrus/client/gui/ClientGUI.py @@ -60,6 +60,7 @@ from hydrus.client.gui import ClientGUIMPV from hydrus.client.gui import ClientGUIPages from hydrus.client.gui import ClientGUIParsing from hydrus.client.gui import ClientGUIPopupMessages +from hydrus.client.gui import ClientGUIScrolledPanels from hydrus.client.gui import ClientGUIScrolledPanelsEdit from hydrus.client.gui import ClientGUIScrolledPanelsManagement from hydrus.client.gui import ClientGUIScrolledPanelsReview @@ -70,6 +71,7 @@ from hydrus.client.gui import ClientGUIStyle from hydrus.client.gui import ClientGUISubscriptions from hydrus.client.gui import ClientGUISystemTray from hydrus.client.gui import ClientGUITags +from hydrus.client.gui import ClientGUITime from hydrus.client.gui import ClientGUITopLevelWindows from hydrus.client.gui import ClientGUITopLevelWindowsPanels from hydrus.client.gui import QtPorting as QP @@ -1154,6 +1156,20 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ): + def _DebugResetColumnListManager( self ): + + message = 'This will reset all saved column widths for all multi-column lists across the program. You may need to restart the client to see changes.' + + result = ClientGUIDialogsQuick.GetYesNo( self, message ) + + if result != QW.QDialog.Accepted: + + return + + + self._controller.column_list_manager.ResetToDefaults() + + def _DebugShowGarbageDifferences( self ): count = collections.Counter() @@ -1430,6 +1446,29 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ): return -1 + def _FixLogicallyInconsistentMappings( self ): + + message = 'This will check for tags that are occupying mutually exclusive states--either current & pending or deleted & petitioned.' + message += os.linesep * 2 + message += 'Please run this if you attempt to upload some tags and get a related error. You may need some follow-up regeneration work to correct autocomplete or \'num pending\' counts.' + + result = ClientGUIDialogsQuick.GetYesNo( self, message, yes_label = 'do it--now choose which service', no_label = 'forget it' ) + + if result == QW.QDialog.Accepted: + + try: + + tag_service_key = GetTagServiceKeyForMaintenance( self ) + + except HydrusExceptions.CancelledException: + + return + + + self._controller.Write( 'fix_logically_inconsistent_mappings', tag_service_key = tag_service_key ) + + + def _FlipClipboardWatcher( self, option_name ): self._controller.new_options.FlipBoolean( option_name ) @@ -2562,6 +2601,74 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ): + def _ManageServiceOptionsUpdatePeriod( self, service_key ): + + service = self._controller.services_manager.GetService( service_key ) + + update_period = service.GetUpdatePeriod() + + with ClientGUITopLevelWindowsPanels.DialogEdit( self, 'edit update period' ) as dlg: + + panel = ClientGUIScrolledPanels.EditSingleCtrlPanel( dlg ) + + height_num_chars = 20 + + control = ClientGUITime.TimeDeltaCtrl( panel, min = HydrusNetwork.MIN_UPDATE_PERIOD, days = True, hours = True, minutes = True, seconds=True ) + + control.SetValue( update_period ) + + panel.SetControl( control ) + + dlg.SetPanel( panel ) + + if dlg.exec() == QW.QDialog.Accepted: + + update_period = control.GetValue() + + if update_period > HydrusNetwork.MAX_UPDATE_PERIOD: + + QW.QMessageBox.information( self, 'Information', 'Sorry, the value you entered was too high. The max is {}.'.format( HydrusData.TimeDeltaToPrettyTimeDelta( HydrusNetwork.MAX_UPDATE_PERIOD ) ) ) + + return + + + job_key = ClientThreading.JobKey() + + job_key.SetVariable( 'popup_title', 'setting update period' ) + job_key.SetVariable( 'popup_text_1', 'uploading\u2026' ) + + self._controller.pub( 'message', job_key ) + + def work_callable(): + + service.Request( HC.POST, 'options_update_period', { 'update_period' : update_period } ) + + return 1 + + + def publish_callable( gumpf ): + + job_key.SetVariable( 'popup_text_1', 'done!' ) + + job_key.Finish() + + service.DoAFullMetadataResync() + + + def errback_ui_cleanup_callable(): + + job_key.SetVariable( 'popup_text_1', 'error!' ) + + job_key.Finish() + + + job = ClientGUIAsync.AsyncQtJob( self, work_callable, publish_callable, errback_ui_cleanup_callable = errback_ui_cleanup_callable ) + + job.start() + + + + def _ManageSubscriptions( self ): def qt_do_it( subscriptions, missing_query_log_container_names, surplus_query_log_container_names, original_pause_status ): @@ -5087,6 +5194,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p ClientGUIMenus.AppendMenuItem( submenu, 'database integrity', 'Have the database examine all its records for internal consistency.', self._CheckDBIntegrity ) ClientGUIMenus.AppendMenuItem( submenu, 'repopulate truncated mappings tables', 'Use the mappings cache to try to repair a previously damaged mappings file.', self._RepopulateMappingsTables ) + ClientGUIMenus.AppendMenuItem( submenu, 'fix logically inconsistent mappings', 'Remove tags that are occupying two mutually exclusive states.', self._FixLogicallyInconsistentMappings ) ClientGUIMenus.AppendMenuItem( submenu, 'fix invalid tags', 'Scan the database for invalid tags.', self._RepairInvalidTags ) ClientGUIMenus.AppendMenu( menu, submenu, 'check and repair' ) @@ -5265,7 +5373,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p ClientGUIMenus.AppendMenuItem( menu, 'review services', 'Look at the services your client connects to.', self._ReviewServices ) ClientGUIMenus.AppendMenuItem( menu, 'manage services', 'Edit the services your client connects to.', self._ManageServices ) - repository_admin_permissions = [ ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_CREATE ), ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_MODERATE ), ( HC.CONTENT_TYPE_ACCOUNT_TYPES, HC.PERMISSION_ACTION_MODERATE ) ] + repository_admin_permissions = [ ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_CREATE ), ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_MODERATE ), ( HC.CONTENT_TYPE_ACCOUNT_TYPES, HC.PERMISSION_ACTION_MODERATE ), ( HC.CONTENT_TYPE_OPTIONS, HC.PERMISSION_ACTION_MODERATE ) ] repositories = self._controller.services_manager.GetServices( HC.REPOSITORIES ) admin_repositories = [ service for service in repositories if True in ( service.HasPermission( content_type, action ) for ( content_type, action ) in repository_admin_permissions ) ] @@ -5285,10 +5393,13 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p service_key = service.GetServiceKey() + service_type = service.GetServiceType() + can_create_accounts = service.HasPermission( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_CREATE ) can_overrule_accounts = service.HasPermission( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_MODERATE ) can_overrule_account_types = service.HasPermission( HC.CONTENT_TYPE_ACCOUNT_TYPES, HC.PERMISSION_ACTION_MODERATE ) can_overrule_services = service.HasPermission( HC.CONTENT_TYPE_SERVICES, HC.PERMISSION_ACTION_MODERATE ) + can_overrule_options = service.HasPermission( HC.CONTENT_TYPE_OPTIONS, HC.PERMISSION_ACTION_MODERATE ) if can_overrule_accounts: @@ -5296,7 +5407,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p ClientGUIMenus.AppendMenuItem( submenu, 'modify an account', 'Modify a specific account\'s type and expiration.', self._ModifyAccount, service_key ) - if can_overrule_accounts and service.GetServiceType() == HC.FILE_REPOSITORY: + if can_overrule_accounts and service_type == HC.FILE_REPOSITORY: ClientGUIMenus.AppendMenuItem( submenu, 'get an uploader\'s ip address', 'Fetch the ip address that uploaded a specific file, if the service knows it.', self._FetchIP, service_key ) @@ -5315,7 +5426,14 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p ClientGUIMenus.AppendMenuItem( submenu, 'manage account types', 'Add, edit and delete account types for this service.', self._STARTManageAccountTypes, service_key ) - if can_overrule_services and service.GetServiceType() == HC.SERVER_ADMIN: + if can_overrule_options and service_type in HC.REPOSITORIES: + + ClientGUIMenus.AppendSeparator( submenu ) + + ClientGUIMenus.AppendMenuItem( submenu, 'change update period', 'Change the update period for this service.', self._ManageServiceOptionsUpdatePeriod, service_key ) + + + if can_overrule_services and service_type == HC.SERVER_ADMIN: ClientGUIMenus.AppendSeparator( submenu ) @@ -5493,6 +5611,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p ClientGUIMenus.AppendMenuItem( gui_actions, 'refresh pages menu in five seconds', 'Delayed refresh the pages menu, giving you time to minimise or otherwise alter the client before it arrives.', self._controller.CallLater, 5, self._menu_updater_pages.update ) ClientGUIMenus.AppendMenuItem( gui_actions, 'publish some sub files in five seconds', 'Publish some files like a subscription would.', self._controller.CallLater, 5, lambda: HG.client_controller.pub( 'imported_files_to_page', [ HydrusData.GenerateKey() for i in range( 5 ) ], 'example sub files' ) ) ClientGUIMenus.AppendMenuItem( gui_actions, 'make a parentless text ctrl dialog', 'Make a parentless text control in a dialog to test some character event catching.', self._DebugMakeParentlessTextCtrl ) + ClientGUIMenus.AppendMenuItem( gui_actions, 'reset multi-column list settings to default', 'Reset all multi-column list widths and other display settings to default.', self._DebugResetColumnListManager ) ClientGUIMenus.AppendMenuItem( gui_actions, 'force a main gui layout now', 'Tell the gui to relayout--useful to test some gui bootup layout issues.', self.adjustSize ) ClientGUIMenus.AppendMenuItem( gui_actions, 'save \'last session\' gui session', 'Make an immediate save of the \'last session\' gui session. Mostly for testing crashes, where last session is not saved correctly.', self.ProposeSaveGUISession, 'last session' ) diff --git a/hydrus/client/gui/ClientGUICanvas.py b/hydrus/client/gui/ClientGUICanvas.py index 33a384d2..551b8c3a 100644 --- a/hydrus/client/gui/ClientGUICanvas.py +++ b/hydrus/client/gui/ClientGUICanvas.py @@ -2183,15 +2183,19 @@ class CanvasWithDetails( Canvas ): # bottom-right index + bottom_right_string = ClientData.ConvertZoomToPercentage( self._current_zoom ) + index_string = self._GetIndexString() if len( index_string ) > 0: - ( text_size, index_string ) = ClientGUIFunctions.GetTextSizeFromPainter( painter, index_string ) - - ClientGUIFunctions.DrawText( painter, my_width - text_size.width() - 3, my_height - text_size.height() - 3, index_string ) + bottom_right_string = '{} - {}'.format( bottom_right_string, index_string ) + ( text_size, bottom_right_string ) = ClientGUIFunctions.GetTextSizeFromPainter( painter, bottom_right_string ) + + ClientGUIFunctions.DrawText( painter, my_width - text_size.width() - 3, my_height - text_size.height() - 3, bottom_right_string ) + def _GetInfoString( self ): diff --git a/hydrus/client/gui/ClientGUIFunctions.py b/hydrus/client/gui/ClientGUIFunctions.py index 6139c09c..fc95d8fb 100644 --- a/hydrus/client/gui/ClientGUIFunctions.py +++ b/hydrus/client/gui/ClientGUIFunctions.py @@ -36,12 +36,19 @@ def ColourIsGreyish( colour: QG.QColor ): return it_is_greyish -def ConvertPixelsToTextWidth( window, pixels ): +def ConvertPixelsToTextWidth( window, pixels, round_down = False ): twenty_chars_in_pixels = int( window.fontMetrics().boundingRect( 20 * 'x' ).width() * MAGIC_TEXT_PADDING ) one_char_in_pixels = twenty_chars_in_pixels / 20 - return round( pixels / one_char_in_pixels ) + if round_down: + + return int( pixels // one_char_in_pixels ) + + else: + + return round( pixels / one_char_in_pixels ) + def ConvertTextToPixels( window, char_dimensions ): diff --git a/hydrus/client/gui/ClientGUIStringPanels.py b/hydrus/client/gui/ClientGUIStringPanels.py index 78bb2c70..ea998050 100644 --- a/hydrus/client/gui/ClientGUIStringPanels.py +++ b/hydrus/client/gui/ClientGUIStringPanels.py @@ -1498,6 +1498,7 @@ class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ): self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT ], ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT ) self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC ], ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC ) + self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_REVERSE ], ClientParsing.CONTENT_PARSER_SORT_TYPE_REVERSE ) tt = 'Human sort sorts numbers as you understand them. "image 2" comes before "image 10". Lexicographic compares each character in turn. "image 02" comes before "image 10", which comes before "image 2".' diff --git a/hydrus/client/gui/lists/ClientGUIListCtrl.py b/hydrus/client/gui/lists/ClientGUIListCtrl.py index 4d6befd6..a57df8c1 100644 --- a/hydrus/client/gui/lists/ClientGUIListCtrl.py +++ b/hydrus/client/gui/lists/ClientGUIListCtrl.py @@ -236,6 +236,12 @@ class BetterListCtrl( QW.QTreeWidget ): last_column_index = num_columns - 1 + # ok, the big pain in the ass situation here is getting a precise last column size that is reproduced on next dialog launch + # ultimately, with fuzzy sizing, style padding, scrollbars appearing, and other weirdness, the more precisely we try to define it, the more we will get dialogs that grow/shrink by a pixel each time + # *therefore*, the actual solution here is to move to snapping with a decent snap distance. the user loses size setting precision, but we'll snap back to a decent size every time, compensating for fuzz + + LAST_COLUMN_SNAP_DISTANCE_CHARS = 5 + for visual_index in range( num_columns ): logical_index = header.logicalIndex( visual_index ) @@ -244,15 +250,22 @@ class BetterListCtrl( QW.QTreeWidget ): width_pixels = header.sectionSize( logical_index ) shown = not header.isSectionHidden( logical_index ) - # if the scrollbar is in place, then when we initialise, next time, we will want to include that extra space in our final column recommended size - # might need to update this to be 'last non-hidden section', rather than 'last 'visual' section' - if visual_index == last_column_index and self.verticalScrollBar().isVisible(): + if visual_index == last_column_index: - width_pixels += self.verticalScrollBar().width() + if self.verticalScrollBar().isVisible(): + + width_pixels += max( 0, min( self.verticalScrollBar().width(), 20 ) ) + width_chars = ClientGUIFunctions.ConvertPixelsToTextWidth( main_tlw, width_pixels ) + if visual_index == last_column_index: + + # here's the snap magic + width_chars = round( width_chars // LAST_COLUMN_SNAP_DISTANCE_CHARS ) * LAST_COLUMN_SNAP_DISTANCE_CHARS + + columns.append( ( column_type, width_chars, shown ) ) @@ -307,7 +320,11 @@ class BetterListCtrl( QW.QTreeWidget ): for i in range( self.topLevelItemCount() ): - if self.topLevelItem( i ).isSelected(): indices.append( i ) + if self.topLevelItem( i ).isSelected(): + + indices.append( i ) + + return indices @@ -747,21 +764,22 @@ class BetterListCtrl( QW.QTreeWidget ): # the issue is: when we first boot up, we want to give a 'hey, it would be nice' size of the last actual recorded final column # HOWEVER, after that: we want to use the current size of the last column # so, if it is the first couple of seconds, lmao. after that, oaml + # I later updated this to use the columnWidth, rather than hickery dickery text-to-pixel-width, since it was juddering resize around text width phase last_column_type = self._column_list_status.GetColumnTypes()[-1] if HydrusData.TimeHasPassed( self._creation_time + 2 ): - last_column_chars = self._column_list_status.GetColumnWidth( last_column_type ) + width += self.columnWidth( self.columnCount() - 1 ) else: last_column_chars = self._original_column_list_status.GetColumnWidth( last_column_type ) - - main_tlw = HG.client_controller.GetMainTLW() - - width += ClientGUIFunctions.ConvertTextToPixelWidth( main_tlw, last_column_chars ) + main_tlw = HG.client_controller.GetMainTLW() + + width += ClientGUIFunctions.ConvertTextToPixelWidth( main_tlw, last_column_chars ) + # diff --git a/hydrus/client/gui/lists/ClientGUIListManager.py b/hydrus/client/gui/lists/ClientGUIListManager.py index f0062f15..b2ad1228 100644 --- a/hydrus/client/gui/lists/ClientGUIListManager.py +++ b/hydrus/client/gui/lists/ClientGUIListManager.py @@ -46,6 +46,13 @@ class ColumnListManager( HydrusSerialisable.SerialisableBase ): return self._column_list_types_to_statuses[ column_list_type ] + def ResetToDefaults( self ): + + self._column_list_types_to_statuses = HydrusSerialisable.SerialisableDictionary() + + self._dirty = True + + def SaveStatus( self, column_list_status: ClientGUIListStatus.ColumnListStatus ): self._column_list_types_to_statuses[ column_list_status.GetColumnListType() ] = column_list_status diff --git a/hydrus/client/gui/search/ClientGUIACDropdown.py b/hydrus/client/gui/search/ClientGUIACDropdown.py index b7076b12..27cd51ee 100644 --- a/hydrus/client/gui/search/ClientGUIACDropdown.py +++ b/hydrus/client/gui/search/ClientGUIACDropdown.py @@ -1991,7 +1991,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ): self._predicates_listbox = ListBoxTagsActiveSearchPredicates( self, self._page_key ) - QP.AddToLayout( self._main_vbox, self._predicates_listbox, CC.FLAGS_EXPAND_PERPENDICULAR ) + QP.AddToLayout( self._main_vbox, self._predicates_listbox, CC.FLAGS_EXPAND_BOTH_WAYS ) def _StartSearchResultsFetchJob( self, job_key ): diff --git a/hydrus/client/gui/search/ClientGUIPredicatesOR.py b/hydrus/client/gui/search/ClientGUIPredicatesOR.py index 5a0cb9be..6e937304 100644 --- a/hydrus/client/gui/search/ClientGUIPredicatesOR.py +++ b/hydrus/client/gui/search/ClientGUIPredicatesOR.py @@ -37,7 +37,7 @@ class ORPredicateControl( QW.QWidget ): vbox = QP.VBoxLayout() - QP.AddToLayout( vbox, self._search_control, CC.FLAGS_CENTER_PERPENDICULAR ) + QP.AddToLayout( vbox, self._search_control, CC.FLAGS_EXPAND_BOTH_WAYS ) self.setLayout( vbox ) diff --git a/hydrus/client/gui/search/ClientGUISearch.py b/hydrus/client/gui/search/ClientGUISearch.py index 4f12c9ef..1c021c79 100644 --- a/hydrus/client/gui/search/ClientGUISearch.py +++ b/hydrus/client/gui/search/ClientGUISearch.py @@ -373,7 +373,16 @@ class EditPredicatesPanel( ClientGUIScrolledPanels.EditPanel ): for panel in self._editable_pred_panels: - QP.AddToLayout( vbox, panel, CC.FLAGS_EXPAND_PERPENDICULAR ) + if isinstance( panel, ClientGUIPredicatesOR.ORPredicateControl ): + + flags = CC.FLAGS_EXPAND_BOTH_WAYS + + else: + + flags = CC.FLAGS_EXPAND_PERPENDICULAR + + + QP.AddToLayout( vbox, panel, flags ) self.widget().setLayout( vbox ) diff --git a/hydrus/client/gui/search/ClientGUISearchPanels.py b/hydrus/client/gui/search/ClientGUISearchPanels.py index f2cb3110..ef9588ee 100644 --- a/hydrus/client/gui/search/ClientGUISearchPanels.py +++ b/hydrus/client/gui/search/ClientGUISearchPanels.py @@ -94,7 +94,7 @@ class EditFavouriteSearchPanel( ClientGUIScrolledPanels.EditPanel ): vbox = QP.VBoxLayout() QP.AddToLayout( vbox, top_gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR ) - QP.AddToLayout( vbox, self._tag_autocomplete, CC.FLAGS_EXPAND_PERPENDICULAR ) + QP.AddToLayout( vbox, self._tag_autocomplete, CC.FLAGS_EXPAND_BOTH_WAYS ) QP.AddToLayout( vbox, bottom_gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR ) self.widget().setLayout( vbox ) diff --git a/hydrus/client/gui/services/ClientGUIClientsideServices.py b/hydrus/client/gui/services/ClientGUIClientsideServices.py index d52d20ae..5cae0141 100644 --- a/hydrus/client/gui/services/ClientGUIClientsideServices.py +++ b/hydrus/client/gui/services/ClientGUIClientsideServices.py @@ -2254,7 +2254,7 @@ class ReviewServiceRestrictedSubPanel( ClientGUICommon.StaticBox ): self._rule_widgets = [] - self._network_sync_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().file_pause, self._PausePlayNetworkSync ) + self._network_sync_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().pause, self._PausePlayNetworkSync ) self._network_sync_paused_button.setToolTip( 'pause/play account sync' ) self._refresh_account_button = ClientGUICommon.BetterButton( self, 'refresh account', self._RefreshAccount ) @@ -2350,11 +2350,11 @@ class ReviewServiceRestrictedSubPanel( ClientGUICommon.StaticBox ): if self._service.IsPausedNetworkSync(): - ClientGUIFunctions.SetBitmapButtonBitmap( self._network_sync_paused_button, CC.global_pixmaps().file_play ) + ClientGUIFunctions.SetBitmapButtonBitmap( self._network_sync_paused_button, CC.global_pixmaps().play ) else: - ClientGUIFunctions.SetBitmapButtonBitmap( self._network_sync_paused_button, CC.global_pixmaps().file_pause ) + ClientGUIFunctions.SetBitmapButtonBitmap( self._network_sync_paused_button, CC.global_pixmaps().pause ) # @@ -2494,10 +2494,10 @@ class ReviewServiceRepositorySubPanel( ClientGUICommon.StaticBox ): self._download_progress = ClientGUICommon.TextAndGauge( self ) - self._update_downloading_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().file_pause, self._PausePlayUpdateDownloading ) + self._update_downloading_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().pause, self._PausePlayUpdateDownloading ) self._update_downloading_paused_button.setToolTip( 'pause/play update downloading' ) - self._update_processing_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().file_pause, self._PausePlayUpdateProcessing ) + self._update_processing_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().pause, self._PausePlayUpdateProcessing ) self._update_processing_paused_button.setToolTip( 'pause/play update processing' ) self._processing_progress = ClientGUICommon.TextAndGauge( self ) @@ -2702,22 +2702,22 @@ class ReviewServiceRepositorySubPanel( ClientGUICommon.StaticBox ): if self._service.IsPausedUpdateDownloading(): - ClientGUIFunctions.SetBitmapButtonBitmap( self._update_downloading_paused_button, CC.global_pixmaps().file_play ) + ClientGUIFunctions.SetBitmapButtonBitmap( self._update_downloading_paused_button, CC.global_pixmaps().play ) else: - ClientGUIFunctions.SetBitmapButtonBitmap( self._update_downloading_paused_button, CC.global_pixmaps().file_pause ) + ClientGUIFunctions.SetBitmapButtonBitmap( self._update_downloading_paused_button, CC.global_pixmaps().pause ) # if self._service.IsPausedUpdateProcessing(): - ClientGUIFunctions.SetBitmapButtonBitmap( self._update_processing_paused_button, CC.global_pixmaps().file_play ) + ClientGUIFunctions.SetBitmapButtonBitmap( self._update_processing_paused_button, CC.global_pixmaps().play ) else: - ClientGUIFunctions.SetBitmapButtonBitmap( self._update_processing_paused_button, CC.global_pixmaps().file_pause ) + ClientGUIFunctions.SetBitmapButtonBitmap( self._update_processing_paused_button, CC.global_pixmaps().pause ) # diff --git a/hydrus/client/networking/ClientLocalServerResources.py b/hydrus/client/networking/ClientLocalServerResources.py index 32001722..89b9a8f3 100644 --- a/hydrus/client/networking/ClientLocalServerResources.py +++ b/hydrus/client/networking/ClientLocalServerResources.py @@ -203,16 +203,33 @@ def ParseClientAPISearchPredicates( request ): request.client_api_permissions.CheckCanSearchTags( tags ) + search_tags = [ ( True, tag ) for tag in tags ] + search_tags.extend( ( ( False, tag ) for tag in negated_tags ) ) + predicates = [] - for tag in negated_tags: + for ( inclusive, tag ) in search_tags: - predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_TAG, value = tag, inclusive = False ) ) + ( namespace, subtag ) = HydrusTags.SplitTag( tag ) - - for tag in tags: + if '*' in tag: + + if subtag == '*': + + tag = namespace + predicate_type = ClientSearch.PREDICATE_TYPE_NAMESPACE + + else: + + predicate_type = ClientSearch.PREDICATE_TYPE_WILDCARD + + + else: + + predicate_type = ClientSearch.PREDICATE_TYPE_TAG + - predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_TAG, value = tag ) ) + predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_TAG, value = tag, inclusive = inclusive ) ) if system_inbox: diff --git a/hydrus/client/networking/ClientNetworkingSessions.py b/hydrus/client/networking/ClientNetworkingSessions.py index ae5a2579..2a6247f4 100644 --- a/hydrus/client/networking/ClientNetworkingSessions.py +++ b/hydrus/client/networking/ClientNetworkingSessions.py @@ -296,6 +296,11 @@ class NetworkSessionManager( HydrusSerialisable.SerialisableBase ): session.verify = False + if not HG.client_controller.new_options.GetBoolean( 'verify_regular_https' ): + + session.verify = False + + return session diff --git a/hydrus/core/HydrusConstants.py b/hydrus/core/HydrusConstants.py index acca47b0..ed10d076 100644 --- a/hydrus/core/HydrusConstants.py +++ b/hydrus/core/HydrusConstants.py @@ -79,8 +79,8 @@ options = {} # Misc NETWORK_VERSION = 20 -SOFTWARE_VERSION = 434 -CLIENT_API_VERSION = 15 +SOFTWARE_VERSION = 435 +CLIENT_API_VERSION = 16 SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 ) @@ -322,6 +322,9 @@ permission_pair_string_lookup[ ( CONTENT_TYPE_ACCOUNTS, PERMISSION_ACTION_MODERA permission_pair_string_lookup[ ( CONTENT_TYPE_ACCOUNT_TYPES, None ) ] = 'cannot change account types' permission_pair_string_lookup[ ( CONTENT_TYPE_ACCOUNT_TYPES, PERMISSION_ACTION_MODERATE ) ] = 'can manage account types completely' +permission_pair_string_lookup[ ( CONTENT_TYPE_OPTIONS, None ) ] = 'cannot change service options' +permission_pair_string_lookup[ ( CONTENT_TYPE_OPTIONS, PERMISSION_ACTION_MODERATE ) ] = 'can manage service options completely' + permission_pair_string_lookup[ ( CONTENT_TYPE_SERVICES, None ) ] = 'cannot change services' permission_pair_string_lookup[ ( CONTENT_TYPE_SERVICES, PERMISSION_ACTION_MODERATE ) ] = 'can manage services completely' diff --git a/hydrus/core/networking/HydrusNetwork.py b/hydrus/core/networking/HydrusNetwork.py index 11bcc360..3636ac76 100644 --- a/hydrus/core/networking/HydrusNetwork.py +++ b/hydrus/core/networking/HydrusNetwork.py @@ -10,6 +10,10 @@ from hydrus.core import HydrusSerialisable from hydrus.core import HydrusTags from hydrus.core.networking import HydrusNetworking +UPDATE_CHECKING_PERIOD = 240 +MIN_UPDATE_PERIOD = 600 +MAX_UPDATE_PERIOD = 100000 * 100 # three months or so jej + def GenerateDefaultServiceDictionary( service_type ): dictionary = HydrusSerialisable.SerialisableDictionary() @@ -109,6 +113,7 @@ def GetPossiblePermissions( service_type ): permissions.append( ( HC.CONTENT_TYPE_ACCOUNTS, [ None, HC.PERMISSION_ACTION_CREATE, HC.PERMISSION_ACTION_MODERATE ] ) ) permissions.append( ( HC.CONTENT_TYPE_ACCOUNT_TYPES, [ None, HC.PERMISSION_ACTION_MODERATE ] ) ) + permissions.append( ( HC.CONTENT_TYPE_OPTIONS, [ None, HC.PERMISSION_ACTION_MODERATE ] ) ) if service_type == HC.FILE_REPOSITORY: @@ -770,7 +775,7 @@ class AccountType( HydrusSerialisable.SerialisableBase ): SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_ACCOUNT_TYPE SERIALISABLE_NAME = 'Account Type' - SERIALISABLE_VERSION = 1 + SERIALISABLE_VERSION = 2 def __init__( self, @@ -852,6 +857,27 @@ class AccountType( HydrusSerialisable.SerialisableBase ): self._auto_creation_history = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_auto_creation_history ) + def _UpdateSerialisableInfo( self, version, old_serialisable_info ): + + if version == 1: + + ( serialisable_account_type_key, title, serialisable_permissions, serialisable_bandwidth_rules, auto_creation_velocity, serialisable_auto_creation_history ) = old_serialisable_info + + permissions = dict( serialisable_permissions ) + + # admins can do options + if HC.CONTENT_TYPE_ACCOUNT_TYPES in permissions and permissions[ HC.CONTENT_TYPE_ACCOUNT_TYPES ] == HC.PERMISSION_ACTION_MODERATE: + + permissions[ HC.CONTENT_TYPE_OPTIONS ] = HC.PERMISSION_ACTION_MODERATE + + + serialisable_permissions = list( permissions.items() ) + + new_serialisable_info = ( serialisable_account_type_key, title, serialisable_permissions, serialisable_bandwidth_rules, auto_creation_velocity, serialisable_auto_creation_history ) + + return ( 2, new_serialisable_info ) + + def BandwidthOK( self, bandwidth_tracker ): return self._bandwidth_rules.CanStartRequest( bandwidth_tracker ) @@ -964,6 +990,7 @@ class AccountType( HydrusSerialisable.SerialisableBase ): permissions[ HC.CONTENT_TYPE_ACCOUNTS ] = HC.PERMISSION_ACTION_MODERATE permissions[ HC.CONTENT_TYPE_ACCOUNT_TYPES ] = HC.PERMISSION_ACTION_MODERATE + permissions[ HC.CONTENT_TYPE_OPTIONS ] = HC.PERMISSION_ACTION_MODERATE if service_type in HC.REPOSITORIES: @@ -1741,8 +1768,6 @@ class Metadata( HydrusSerialisable.SerialisableBase ): SERIALISABLE_NAME = 'Metadata' SERIALISABLE_VERSION = 1 - CLIENT_DELAY = 20 * 60 - def __init__( self, metadata = None, next_update_due = None ): if metadata is None: @@ -1766,6 +1791,24 @@ class Metadata( HydrusSerialisable.SerialisableBase ): self._update_hashes = set() + self._biggest_end = self._CalculateBiggestEnd() + + + def _CalculateBiggestEnd( self ): + + if len( self._metadata ) == 0: + + return None + + else: + + biggest_index = max( self._metadata.keys() ) + + ( update_hashes, begin, end ) = self._GetUpdate( biggest_index ) + + return end + + def _GetNextUpdateDueTime( self, from_client = False ): @@ -1773,7 +1816,7 @@ class Metadata( HydrusSerialisable.SerialisableBase ): if from_client: - delay = self.CLIENT_DELAY + delay = UPDATE_CHECKING_PERIOD * 2 return self._next_update_due + delay @@ -1818,6 +1861,8 @@ class Metadata( HydrusSerialisable.SerialisableBase ): self._update_hashes.update( update_hashes ) + self._biggest_end = self._CalculateBiggestEnd() + def AppendUpdate( self, update_hashes, begin, end, next_update_due ): @@ -1831,6 +1876,23 @@ class Metadata( HydrusSerialisable.SerialisableBase ): self._next_update_due = next_update_due + self._biggest_end = end + + + + def CalculateNewNextUpdateDue( self, update_period ): + + with self._lock: + + if self._biggest_end is None: + + self._next_update_due = 0 + + else: + + self._next_update_due = self._biggest_end + update_period + + def GetEarliestTimestampForTheseHashes( self, hashes ): @@ -1865,19 +1927,13 @@ class Metadata( HydrusSerialisable.SerialisableBase ): with self._lock: - keys = list( self._metadata.keys() ) - - if len( keys ) == 0: + if self._biggest_end is None: return HydrusData.GetNow() else: - largest_update_index = max( keys ) - - ( update_hashes, begin, end ) = self._GetUpdate( largest_update_index ) - - return end + 1 + return self._biggest_end + 1 @@ -1890,20 +1946,24 @@ class Metadata( HydrusSerialisable.SerialisableBase ): return 'have not yet synced metadata' + elif self._biggest_end is None: + + return 'the metadata appears to be uninitialised' + else: update_due = self._GetNextUpdateDueTime( from_client ) if HydrusData.TimeHasPassed( update_due ): - s = 'imminently' + s = 'next update imminent' else: - s = HydrusData.TimestampToPrettyTimeDelta( update_due ) + s = 'next update due {}'.format( HydrusData.TimestampToPrettyTimeDelta( update_due ) ) - return 'next update due ' + s + return 'metadata synced up to {}, {}'.format( HydrusData.TimestampToPrettyTimeDelta( self._biggest_end ), s ) @@ -1982,45 +2042,6 @@ class Metadata( HydrusSerialisable.SerialisableBase ): - def GetUpdateInfo( self, from_client = False ): - - with self._lock: - - num_update_hashes = sum( ( len( update_hashes ) for ( update_hashes, begin, end ) in list(self._metadata.values()) ) ) - - if len( self._metadata ) == 0: - - status = 'have not yet synchronised' - - else: - - biggest_end = max( ( end for ( update_hashes, begin, end ) in list(self._metadata.values()) ) ) - - delay = 0 - - if from_client: - - delay = self.CLIENT_DELAY - - - next_update_time = self._next_update_due + delay - - if HydrusData.TimeHasPassed( next_update_time ): - - s = 'imminently' - - else: - - s = HydrusData.TimestampToPrettyTimeDelta( next_update_time ) - - - status = 'metadata synchronised up to ' + HydrusData.TimestampToPrettyTimeDelta( biggest_end ) + ', next update due ' + s - - - return ( num_update_hashes, status ) - - - def HasDoneInitialSync( self ): with self._lock: @@ -2037,6 +2058,15 @@ class Metadata( HydrusSerialisable.SerialisableBase ): + def UpdateASAP( self ): + + with self._lock: + + # not 0, that's reserved + self._next_update_due = 1 + + + def UpdateDue( self, from_client = False ): with self._lock: @@ -2054,6 +2084,7 @@ class Metadata( HydrusSerialisable.SerialisableBase ): self._metadata.update( metadata_slice._metadata ) self._next_update_due = metadata_slice._next_update_due + self._biggest_end = self._CalculateBiggestEnd() @@ -2477,6 +2508,20 @@ class ServerServiceRepository( ServerServiceRestricted ): + def SetUpdatePeriod( self, update_period: int ): + + with self._lock: + + self._service_options[ 'update_period' ] = update_period + + self._metadata.CalculateNewNextUpdateDue( update_period ) + + self._SetDirty() + + HG.server_controller.pub( 'notify_new_repo_sync' ) + + + def Sync( self ): with self._lock: diff --git a/hydrus/server/ServerController.py b/hydrus/server/ServerController.py index 95a28e6b..ab4d3be3 100644 --- a/hydrus/server/ServerController.py +++ b/hydrus/server/ServerController.py @@ -13,6 +13,7 @@ from hydrus.core import HydrusExceptions from hydrus.core import HydrusGlobals as HG from hydrus.core import HydrusSessions from hydrus.core import HydrusThreading +from hydrus.core.networking import HydrusNetwork from hydrus.core.networking import HydrusNetworking from hydrus.server import ServerDB @@ -257,7 +258,8 @@ class Controller( HydrusController.HydrusController ): # - job = self.CallRepeating( 5.0, 600.0, self.SyncRepositories ) + job = self.CallRepeating( 5.0, HydrusNetwork.UPDATE_CHECKING_PERIOD, self.SyncRepositories ) + job.WakeOnPubSub( 'notify_new_repo_sync' ) self._daemon_jobs[ 'sync_repositories' ] = job diff --git a/hydrus/server/ServerDB.py b/hydrus/server/ServerDB.py index d13cc195..dd335408 100644 --- a/hydrus/server/ServerDB.py +++ b/hydrus/server/ServerDB.py @@ -1574,12 +1574,6 @@ class DB( HydrusDB.HydrusDB ): self._c.execute( 'INSERT OR IGNORE INTO ' + current_tag_parents_table_name + ' ( child_service_tag_id, parent_service_tag_id, account_id, parent_timestamp ) VALUES ( ?, ?, ?, ? );', ( child_service_tag_id, parent_service_tag_id, account_id, timestamp ) ) - child_master_hash_ids = self._RepositoryGetCurrentMappingsMasterHashIds( service_id, child_service_tag_id ) - - overwrite_deleted = False - - self._RepositoryAddMappings( service_id, account_id, parent_master_tag_id, child_master_hash_ids, overwrite_deleted, timestamp ) - def _RepositoryAddTagSibling( self, service_id, account_id, bad_master_tag_id, good_master_tag_id, overwrite_deleted, timestamp ): @@ -2070,17 +2064,6 @@ class DB( HydrusDB.HydrusDB ): return count - def _RepositoryGetCurrentMappingsMasterHashIds( self, service_id, service_tag_id ): - - ( hash_id_map_table_name, tag_id_map_table_name ) = GenerateRepositoryMasterMapTableNames( service_id ) - - ( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateRepositoryMappingsTableNames( service_id ) - - master_hash_ids = [ master_hash_id for ( master_hash_id, ) in self._c.execute( 'SELECT master_hash_id FROM ' + hash_id_map_table_name + ' NATURAL JOIN ' + current_mappings_table_name + ' WHERE service_tag_id = ?;', ( service_tag_id, ) ) ] - - return master_hash_ids - - def _RepositoryGetFilesInfoFilesTableJoin( self, service_id, content_status ): ( hash_id_map_table_name, tag_id_map_table_name ) = GenerateRepositoryMasterMapTableNames( service_id ) diff --git a/hydrus/server/networking/ServerServer.py b/hydrus/server/networking/ServerServer.py index bd1c51c5..9657ad42 100644 --- a/hydrus/server/networking/ServerServer.py +++ b/hydrus/server/networking/ServerServer.py @@ -30,6 +30,8 @@ class HydrusServiceRestricted( HydrusServer.HydrusService ): root.putChild( b'account_types', ServerServerResources.HydrusResourceRestrictedAccountTypes( self._service, HydrusServer.REMOTE_DOMAIN ) ) + root.putChild( b'options_update_period', ServerServerResources.HydrusResourceRestrictedOptionsModifyUpdatePeriod( self._service, HydrusServer.REMOTE_DOMAIN ) ) + root.putChild( b'registration_keys', ServerServerResources.HydrusResourceRestrictedRegistrationKeys( self._service, HydrusServer.REMOTE_DOMAIN ) ) return root diff --git a/hydrus/server/networking/ServerServerResources.py b/hydrus/server/networking/ServerServerResources.py index f781a094..ba4838c7 100644 --- a/hydrus/server/networking/ServerServerResources.py +++ b/hydrus/server/networking/ServerServerResources.py @@ -382,6 +382,36 @@ class HydrusResourceRestrictedOptions( HydrusResourceRestricted ): return response_context +class HydrusResourceRestrictedOptionsModify( HydrusResourceRestricted ): + + def _checkAccountPermissions( self, request: HydrusServerRequest.HydrusRequest ): + + request.hydrus_account.CheckPermission( HC.CONTENT_TYPE_OPTIONS, HC.PERMISSION_ACTION_MODERATE ) + + +class HydrusResourceRestrictedOptionsModifyUpdatePeriod( HydrusResourceRestrictedOptionsModify ): + + def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ): + + update_period = request.parsed_request_args[ 'update_period' ] + + if update_period < HydrusNetwork.MIN_UPDATE_PERIOD: + + raise HydrusExceptions.BadRequestException( 'The update period was too low. It needs to be at least {}.'.format( HydrusData.TimeDeltaToPrettyTimeDelta( HydrusNetwork.MIN_UPDATE_PERIOD ) ) ) + + + if update_period > HydrusNetwork.MAX_UPDATE_PERIOD: + + raise HydrusExceptions.BadRequestException( 'The update period was too high. It needs to be lower than {}.'.format( HydrusData.TimeDeltaToPrettyTimeDelta( HydrusNetwork.MAX_UPDATE_PERIOD ) ) ) + + + self._service.SetUpdatePeriod( update_period ) + + response_context = HydrusServerResources.ResponseContext( 200 ) + + return response_context + + class HydrusResourceRestrictedAccountModify( HydrusResourceRestricted ): def _checkAccountPermissions( self, request: HydrusServerRequest.HydrusRequest ):
-
+
There is now a simple 'session' system, where you can get a temporary key that gives the same access without having to include the permanent access key in every request. You can fetch a session key with the /session_key command and thereafter use it just as you would an access key, just with Hydrus-Client-API-Session-Key instead.
Session keys will expire if they are not used within 24 hours, or if the client is restarted, or if the underlying access key is deleted. An invalid/expired session key will give a 419 result with an appropriate error text.
-Bear in mind the Client API is still under construction and is http-only for the moment--be careful about transmitting sensitive content outside of localhost. The access key will be unencrypted across any connection, and if it is included as a GET parameter, as simple and convenient as that is, it could be cached in all sorts of places.
+Bear in mind the Client API is still under construction. Setting up the Client API accessible across the internet requires technical experience to be convenient. HTTPS is available for encrypted comms, but the default certificate is self-signed (which basically means an eavesdropper can't see through it, but your ISP/government could if they decided to target you). If you have your own domain and SSL cert, you can replace them though (check the db directory for client.crt and client.key). Otherwise, be careful about transmitting sensitive content outside of your localhost/network.
Contents
If the access key's permissions only permit search for certain tags, at least one whitelisted/non-blacklisted tag must be in the "tags" list or this will 403. Tags can be prepended with a hyphen to make a negated tag (e.g. "-green eyes"), but these will not be eligible for the permissions whitelist check.
+If the access key's permissions only permit search for certain tags, at least one whitelisted/non-blacklisted tag must be in the "tags" list or this will 403. Tags can be prepended with a hyphen to make a negated tag (e.g. "-green eyes"), but these will not be checked against the permissions whitelist.
+Wildcards and namespace searches are now supported, so if you search for 'character:sam*' or 'series:*', this will be handled correctly clientside.
Response description: The full list of numerical file ids that match the search.