Version 430

closes #799, closes #815

help/changelog.html

@@ -8,6 +8,33 @@
 		<div class="content">
 			<h3 id="changelog"><a href="#changelog">changelog</a></h3>
 			<ul>
+				<li><h3 id="version_430"><a href="#version_430">version 430</a></h3></li>
+				<ul>
+					<li>misc:</li>
+					<li>fixed 'unusual character' collapse logic for short text inputs in tag autocomplete lookups. in human, this means typing 'a' now correctly gives you the tag '/a/' and _vice versa_ (issue #799)</li>
+					<li>to make this work, an old database subtag map cache is revived this week in a more efficient form. if you sync with the PTR, it will take a couple minutes to update. the regen routine is also added to the database->regen menu, in case it ever desynchronises in future</li>
+					<li>absent an override referral url, api-linked url fetches now use the original url as referrer. previously they were sending no referrer. this fixes watching spicy boards on 8chan.moe</li>
+					<li>updated a 'get all this stuff' database routine to report more info, and a handful of supermassive jobs (mostly db maintenance regen) now report x/y progress with y, rather than just a nebulous increasing x</li>
+					<li>fixed an odd bug in a common UI text-clearing call that was causing real text not to show up for a while after the clear. this was most apparent in the downloader highlight panels, where status text on file/gallery/network status could sometimes stay blank until a change</li>
+					<li>the manage tags dialog's "there are several things you can do" button box when you enter tags in complicated situations is now clearer. there are several sorts of intro text on the dialog, the button labels are clearer, and button tooltips have more action information</li>
+					<li>fixed the tumblr downloader! sorry for the trouble here, I hadn't realised the situation from some reports. if you have tumblr subs, please go into them and set to 'try again' any recent urls that say 'Found 0 new URLs.'</li>
+					<li>.</li>
+					<li>taglists:</li>
+					<li>you can now right-click any edit/write taglist (like those across the manage tags dialog) and choose to hide/show the implied parents that now hang underneath tags</li>
+					<li>you can set whether this defaults to hide or show, separately for the regular taglists and the autocomplete results dropdown, under options->tags</li>
+					<li>the taglist now sorts lexicographically using sibling tag data where available. I had expected to make options here to use storage or ideal tag, but once I tried it out, using the ideal all the time felt proper to me, so let's see how it goes</li>
+					<li>fixed the routine that removes mutually exclusive predicates (e.g. system:inbox/archive) when adding to the active search predicates taglist. this fixes the 'exclude xxx from search' menu action and other add/swap actions (issue #815)</li>
+					<li>gave the taglist right-click menu another quick pass. since there are all sorts of actions that may or not appear, and menu items can get pretty wide with tag text, I am trying out an intentionally short and thin top-level menu of 'verbs' that is quick to navigate with your mouse, and then tuck longer and taller stuff in secondary menus</li>
+					<li>.</li>
+					<li>boring code cleanup:</li>
+					<li>cleaned and unified a bunch of the new taglist sibling and parents display logic and other legacy variables. it now basically all derives from one storage/display state, so behaviour across the program should be more unified. this may cause confusion in some more advanced dialogs, so let me know anywhere it looks weird</li>
+					<li>the 'favourites' autocomplete tab in 'edit/write' a/c dropdowns now show siblings and parents for the current display service</li>
+					<li>the tag suggestions favourites dropdowns and taglists in the options now show siblings/parents according to the current service</li>
+					<li>the 'url class precedence' routine, which tests more 'specific' url classes first when trying to match an url, has a subtle logic change--now, url classes are first considered more 'specific' according to number of path components and parameters that have no default. this stops an url class with multiple optional parameters overriding another with a single fixed parameter (this is what affected the tumblr downloader above). the specific (descending) sort key is now (required components, total components, required parameters, total parameters, len normalised example url)</li>
+					<li>refactored client object serialisation access routines to a new db module</li>
+					<li>refactored database transaction code and status tracking to a separate object</li>
+					<li>refactored some more tag definition routines to the master tag module</li>
+				</ul>
 				<li><h3 id="version_429"><a href="#version_429">version 429</a></h3></li>
 				<ul>
 					<li>misc:</li>

hydrus/client/ClientExporting.py

@@ -16,6 +16,7 @@ from hydrus.client import ClientPaths
 from hydrus.client import ClientSearch
 from hydrus.client.media import ClientMediaManagers
 from hydrus.client.metadata import ClientTags
+from hydrus.client.metadata import ClientTagSorting
 
 MAX_PATH_LENGTH = 240 # bit of padding from 255 for .txt neigbouring and other surprises
 
@@ -643,7 +644,7 @@ class SidecarExporter( HydrusSerialisable.SerialisableBase ):
             
             all_tags = list( all_tags )
             
-            ClientTags.SortTags( CC.SORT_BY_LEXICOGRAPHIC_DESC, all_tags )
+            ClientTagSorting.SortTags( CC.SORT_BY_LEXICOGRAPHIC_DESC, all_tags )
             
             txt_path = os.path.join( directory, filename + '.txt' )
             

hydrus/client/ClientOptions.py

@@ -229,6 +229,9 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
         
         self._dictionary[ 'booleans' ][ 'notify_client_api_cookies' ] = False
         
+        self._dictionary[ 'booleans' ][ 'expand_parents_on_storage_taglists' ] = True
+        self._dictionary[ 'booleans' ][ 'expand_parents_on_storage_autocomplete_taglists' ] = True
+        
         #
         
         self._dictionary[ 'colours' ] = HydrusSerialisable.SerialisableDictionary()

hydrus/client/db/ClientDBMaster.py

@@ -6,7 +6,6 @@ from hydrus.core import HydrusData
 from hydrus.core import HydrusDB
 from hydrus.core import HydrusDBModule
 from hydrus.core import HydrusExceptions
-from hydrus.core import HydrusSerialisable
 from hydrus.core import HydrusTags
 
 from hydrus.client.networking import ClientNetworkingDomain
@@ -580,6 +579,100 @@ class ClientDBMasterTags( HydrusDBModule.HydrusDBModule ):
         return tag_ids_to_tags
         
     
+    def NamespaceExists( self, namespace ):
+        
+        if namespace == '':
+            
+            return True
+            
+        
+        result = self._c.execute( 'SELECT 1 FROM namespaces WHERE namespace = ?;', ( namespace, ) ).fetchone()
+        
+        if result is None:
+            
+            return False
+            
+        else:
+            
+            return True
+            
+        
+    
+    def SubtagExists( self, subtag ):
+        
+        try:
+            
+            HydrusTags.CheckTagNotEmpty( subtag )
+            
+        except HydrusExceptions.TagSizeException:
+            
+            return False
+            
+        
+        result = self._c.execute( 'SELECT 1 FROM subtags WHERE subtag = ?;', ( subtag, ) ).fetchone()
+        
+        if result is None:
+            
+            return False
+            
+        else:
+            
+            return True
+            
+        
+    
+    def TagExists( self, tag ):
+        
+        try:
+            
+            tag = HydrusTags.CleanTag( tag )
+            
+        except:
+            
+            return False
+            
+        
+        try:
+            
+            HydrusTags.CheckTagNotEmpty( tag )
+            
+        except HydrusExceptions.TagSizeException:
+            
+            return False
+            
+        
+        ( namespace, subtag ) = HydrusTags.SplitTag( tag )
+        
+        if self.NamespaceExists( namespace ):
+            
+            namespace_id = self.GetNamespaceId( namespace )
+            
+        else:
+            
+            return False
+            
+        
+        if self.SubtagExists( subtag ):
+            
+            subtag_id = self.GetSubtagId( subtag )
+            
+            result = self._c.execute( 'SELECT 1 FROM tags WHERE namespace_id = ? AND subtag_id = ?;', ( namespace_id, subtag_id ) ).fetchone()
+            
+            if result is None:
+                
+                return False
+                
+            else:
+                
+                return True
+                
+            
+        else:
+            
+            return False
+            
+        
+    
     def UpdateTagId( self, tag_id, namespace_id, subtag_id ):
         
         self._c.execute( 'UPDATE tags SET namespace_id = ?, subtag_id = ? WHERE tag_id = ?;', ( namespace_id, subtag_id, tag_id ) )

hydrus/client/db/ClientDBSerialisable.py

@@ -0,0 +1,644 @@
+import json
+import os
+import sqlite3
+import time
+import typing
+
+from hydrus.core import HydrusConstants as HC
+from hydrus.core import HydrusData
+from hydrus.core import HydrusDB
+from hydrus.core import HydrusDBModule
+from hydrus.core import HydrusExceptions
+from hydrus.core import HydrusGlobals as HG
+from hydrus.core import HydrusSerialisable
+
+from hydrus.client import ClientConstants as CC
+from hydrus.client.db import ClientDBServices
+
+YAML_DUMP_ID_SINGLE = 0
+YAML_DUMP_ID_REMOTE_BOORU = 1
+YAML_DUMP_ID_FAVOURITE_CUSTOM_FILTER_ACTIONS = 2
+YAML_DUMP_ID_GUI_SESSION = 3
+YAML_DUMP_ID_IMAGEBOARD = 4
+YAML_DUMP_ID_IMPORT_FOLDER = 5
+YAML_DUMP_ID_EXPORT_FOLDER = 6
+YAML_DUMP_ID_SUBSCRIPTION = 7
+YAML_DUMP_ID_LOCAL_BOORU = 8
+
+def DealWithBrokenJSONDump( db_dir, dump, dump_descriptor ):
+    
+    timestamp_string = time.strftime( '%Y-%m-%d %H-%M-%S' )
+    hex_chars = os.urandom( 4 ).hex()
+    
+    filename = '({}) at {} {}.json'.format( dump_descriptor, timestamp_string, hex_chars )
+    
+    path = os.path.join( db_dir, filename )
+    
+    with open( path, 'wb' ) as f:
+        
+        if isinstance( dump, str ):
+            
+            dump = bytes( dump, 'utf-8', errors = 'replace' )
+            
+        
+        f.write( dump )
+        
+    
+    message = 'A serialised object failed to load! Its description is "{}".'.format( dump_descriptor )
+    message += os.linesep * 2
+    message += 'This error could be due to several factors, but is most likely a hard drive fault (perhaps your computer recently had a bad power cut?).'
+    message += os.linesep * 2
+    message += 'The database has attempted to delete the broken object, errors have been written to the log, and the object\'s dump written to {}. Depending on the object, your client may no longer be able to boot, or it may have lost something like a session or a subscription.'.format( path )
+    message += os.linesep * 2
+    message += 'Please review the \'help my db is broke.txt\' file in your install_dir/db directory as background reading, and if the situation or fix here is not obvious, please contact hydrus dev.'
+    
+    HydrusData.ShowText( message )
+    
+    raise HydrusExceptions.SerialisationException( message )
+    
+def GenerateBigSQLiteDumpBuffer( dump ):
+    
+    try:
+        
+        dump_bytes = bytes( dump, 'utf-8' )
+        
+    except Exception as e:
+        
+        HydrusData.PrintException( e )
+        
+        raise Exception( 'While trying to save data to the database, it could not be decoded from UTF-8 to bytes! This could indicate an encoding error, such as Shift JIS sneaking into a downloader page! Please let hydrus dev know about this! Full error was written to the log!' )
+        
+    
+    if len( dump_bytes ) >= 1073741824: # 1GB
+        
+        raise Exception( 'A data object could not save to the database because it was bigger than a buffer limit of 1GB! If your session has hundreds of thousands of files or URLs in it, close some pages NOW! Otherwise, please report this to hydrus dev!' )
+        
+    
+    try:
+        
+        dump_buffer = sqlite3.Binary( dump_bytes )
+        
+    except Exception as e:
+        
+        HydrusData.PrintException( e )
+        
+        raise Exception( 'While trying to save data to the database, it would not form into a buffer! Please let hydrus dev know about this! Full error was written to the log!' )
+        
+    
+    return dump_buffer
+    
+class ClientDBSerialisable( HydrusDBModule.HydrusDBModule ):
+    
+    def __init__( self, cursor: sqlite3.Cursor, db_dir, cursor_transaction_wrapper: HydrusDB.DBCursorTransactionWrapper, modules_services: ClientDBServices.ClientDBMasterServices ):
+        
+        HydrusDBModule.HydrusDBModule.__init__( self, 'client serialisable', cursor )
+        
+        self._db_dir = db_dir
+        self._cursor_transaction_wrapper = cursor_transaction_wrapper
+        self.modules_services = modules_services
+        
+    
+    def _GetInitialIndexGenerationTuples( self ):
+        
+        index_generation_tuples = []
+        
+        return index_generation_tuples
+        
+    
+    def CreateInitialTables( self ):
+        
+        self._c.execute( 'CREATE TABLE json_dict ( name TEXT PRIMARY KEY, dump BLOB_BYTES );' )
+        self._c.execute( 'CREATE TABLE json_dumps ( dump_type INTEGER PRIMARY KEY, version INTEGER, dump BLOB_BYTES );' )
+        self._c.execute( 'CREATE TABLE json_dumps_named ( dump_type INTEGER, dump_name TEXT, version INTEGER, timestamp INTEGER, dump BLOB_BYTES, PRIMARY KEY ( dump_type, dump_name, timestamp ) );' )
+        
+        self._c.execute( 'CREATE TABLE yaml_dumps ( dump_type INTEGER, dump_name TEXT, dump TEXT_YAML, PRIMARY KEY ( dump_type, dump_name ) );' )
+        
+    
+    def DeleteJSONDump( self, dump_type ):
+        
+        self._c.execute( 'DELETE FROM json_dumps WHERE dump_type = ?;', ( dump_type, ) )
+        
+    
+    def DeleteJSONDumpNamed( self, dump_type, dump_name = None, timestamp = None ):
+        
+        if dump_name is None:
+            
+            self._c.execute( 'DELETE FROM json_dumps_named WHERE dump_type = ?;', ( dump_type, ) )
+            
+        elif timestamp is None:
+            
+            self._c.execute( 'DELETE FROM json_dumps_named WHERE dump_type = ? AND dump_name = ?;', ( dump_type, dump_name ) )
+            
+        else:
+            
+            self._c.execute( 'DELETE FROM json_dumps_named WHERE dump_type = ? AND dump_name = ? AND timestamp = ?;', ( dump_type, dump_name, timestamp ) )
+            
+        
+    
+    def DeleteYAMLDump( self, dump_type, dump_name = None ):
+        
+        if dump_name is None:
+            
+            self._c.execute( 'DELETE FROM yaml_dumps WHERE dump_type = ?;', ( dump_type, ) )
+            
+        else:
+            
+            if dump_type == YAML_DUMP_ID_LOCAL_BOORU: dump_name = dump_name.hex()
+            
+            self._c.execute( 'DELETE FROM yaml_dumps WHERE dump_type = ? AND dump_name = ?;', ( dump_type, dump_name ) )
+            
+        
+        if dump_type == YAML_DUMP_ID_LOCAL_BOORU:
+            
+            service_id = self.modules_services.GetServiceId( CC.LOCAL_BOORU_SERVICE_KEY )
+            
+            self._c.execute( 'DELETE FROM service_info WHERE service_id = ? AND info_type = ?;', ( service_id, HC.SERVICE_INFO_NUM_SHARES ) )
+            
+            HG.client_controller.pub( 'refresh_local_booru_shares' )
+            
+        
+    
+    def GetExpectedTableNames( self ) -> typing.Collection[ str ]:
+        
+        expected_table_names = [
+            'json_dict',
+            'json_dumps',
+            'json_dumps_named',
+            'yaml_dumps'
+        ]
+        
+        return expected_table_names
+        
+    
+    def GetJSONDump( self, dump_type ):
+        
+        result = self._c.execute( 'SELECT version, dump FROM json_dumps WHERE dump_type = ?;', ( dump_type, ) ).fetchone()
+        
+        if result is None:
+            
+            return result
+            
+        else:
+            
+            ( version, dump ) = result
+            
+            try:
+                
+                if isinstance( dump, bytes ):
+                    
+                    dump = str( dump, 'utf-8' )
+                    
+                
+                serialisable_info = json.loads( dump )
+                
+            except:
+                
+                self._c.execute( 'DELETE FROM json_dumps WHERE dump_type = ?;', ( dump_type, ) )
+                
+                self._cursor_transaction_wrapper.CommitAndBegin()
+                
+                DealWithBrokenJSONDump( self._db_dir, dump, 'dump_type {}'.format( dump_type ) )
+                
+            
+            obj = HydrusSerialisable.CreateFromSerialisableTuple( ( dump_type, version, serialisable_info ) )
+            
+            if dump_type == HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER:
+                
+                session_containers = self.GetJSONDumpNamed( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER_SESSION_CONTAINER )
+                
+                obj.SetSessionContainers( session_containers )
+                
+            elif dump_type == HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_BANDWIDTH_MANAGER:
+                
+                tracker_containers = self.GetJSONDumpNamed( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_BANDWIDTH_MANAGER_TRACKER_CONTAINER )
+                
+                obj.SetTrackerContainers( tracker_containers )
+                
+            
+            return obj
+            
+        
+    
+    def GetJSONDumpNamed( self, dump_type, dump_name = None, timestamp = None ):
+        
+        if dump_name is None:
+            
+            results = self._c.execute( 'SELECT dump_name, version, dump, timestamp FROM json_dumps_named WHERE dump_type = ?;', ( dump_type, ) ).fetchall()
+            
+            objs = []
+            
+            for ( dump_name, version, dump, object_timestamp ) in results:
+                
+                try:
+                    
+                    if isinstance( dump, bytes ):
+                        
+                        dump = str( dump, 'utf-8' )
+                        
+                    
+                    serialisable_info = json.loads( dump )
+                    
+                    objs.append( HydrusSerialisable.CreateFromSerialisableTuple( ( dump_type, dump_name, version, serialisable_info ) ) )
+                    
+                except:
+                    
+                    self._c.execute( 'DELETE FROM json_dumps_named WHERE dump_type = ? AND dump_name = ? AND timestamp = ?;', ( dump_type, dump_name, object_timestamp ) )
+                    
+                    self._cursor_transaction_wrapper.CommitAndBegin()
+                    
+                    DealWithBrokenJSONDump( self._db_dir, dump, 'dump_type {} dump_name {} timestamp {}'.format( dump_type, dump_name[:10], timestamp ) )
+                    
+                
+            
+            return objs
+            
+        else:
+            
+            if timestamp is None:
+                
+                result = self._c.execute( 'SELECT version, dump, timestamp FROM json_dumps_named WHERE dump_type = ? AND dump_name = ? ORDER BY timestamp DESC;', ( dump_type, dump_name ) ).fetchone()
+                
+            else:
+                
+                result = self._c.execute( 'SELECT version, dump, timestamp FROM json_dumps_named WHERE dump_type = ? AND dump_name = ? AND timestamp = ?;', ( dump_type, dump_name, timestamp ) ).fetchone()
+                
+            
+            if result is None:
+                
+                raise HydrusExceptions.DataMissing( 'Could not find the object of type "{}" and name "{}" and timestamp "{}".'.format( dump_type, dump_name, str( timestamp ) ) )
+                
+            
+            ( version, dump, object_timestamp ) = result
+            
+            try:
+                
+                if isinstance( dump, bytes ):
+                    
+                    dump = str( dump, 'utf-8' )
+                    
+                
+                serialisable_info = json.loads( dump )
+                
+            except:
+                
+                self._c.execute( 'DELETE FROM json_dumps_named WHERE dump_type = ? AND dump_name = ? AND timestamp = ?;', ( dump_type, dump_name, object_timestamp ) )
+                
+                self._cursor_transaction_wrapper.CommitAndBegin()
+                
+                DealWithBrokenJSONDump( self._db_dir, dump, 'dump_type {} dump_name {} timestamp {}'.format( dump_type, dump_name[:10], object_timestamp ) )
+                
+            
+            return HydrusSerialisable.CreateFromSerialisableTuple( ( dump_type, dump_name, version, serialisable_info ) )
+            
+        
+    
+    def GetJSONDumpNames( self, dump_type ):
+        
+        names = [ name for ( name, ) in self._c.execute( 'SELECT DISTINCT dump_name FROM json_dumps_named WHERE dump_type = ?;', ( dump_type, ) ) ]
+        
+        return names
+        
+    
+    def GetJSONDumpNamesToBackupTimestamps( self, dump_type ):
+        
+        names_to_backup_timestamps = HydrusData.BuildKeyToListDict( self._c.execute( 'SELECT dump_name, timestamp FROM json_dumps_named WHERE dump_type = ? ORDER BY timestamp ASC;', ( dump_type, ) ) )
+        
+        for ( name, timestamp_list ) in list( names_to_backup_timestamps.items() ):
+            
+            timestamp_list.pop( -1 ) # remove the non backup timestamp
+            
+            if len( timestamp_list ) == 0:
+                
+                del names_to_backup_timestamps[ name ]
+                
+            
+        
+        return names_to_backup_timestamps
+        
+    
+    def GetJSONSimple( self, name ):
+        
+        result = self._c.execute( 'SELECT dump FROM json_dict WHERE name = ?;', ( name, ) ).fetchone()
+        
+        if result is None:
+            
+            return None
+            
+        
+        ( dump, ) = result
+        
+        if isinstance( dump, bytes ):
+            
+            dump = str( dump, 'utf-8' )
+            
+        
+        value = json.loads( dump )
+        
+        return value
+        
+    
+    def GetYAMLDump( self, dump_type, dump_name = None ):
+        
+        if dump_name is None:
+            
+            result = { dump_name : data for ( dump_name, data ) in self._c.execute( 'SELECT dump_name, dump FROM yaml_dumps WHERE dump_type = ?;', ( dump_type, ) ) }
+            
+            if dump_type == YAML_DUMP_ID_LOCAL_BOORU:
+                
+                result = { bytes.fromhex( dump_name ) : data for ( dump_name, data ) in list(result.items()) }
+                
+            
+        else:
+            
+            if dump_type == YAML_DUMP_ID_LOCAL_BOORU: dump_name = dump_name.hex()
+            
+            result = self._c.execute( 'SELECT dump FROM yaml_dumps WHERE dump_type = ? AND dump_name = ?;', ( dump_type, dump_name ) ).fetchone()
+            
+            if result is None:
+                
+                if result is None:
+                    
+                    raise HydrusExceptions.DataMissing( dump_name + ' was not found!' )
+                    
+                
+            else:
+                
+                ( result, ) = result
+                
+            
+        
+        return result
+        
+    
+    def GetYAMLDumpNames( self, dump_type ):
+        
+        names = [ name for ( name, ) in self._c.execute( 'SELECT dump_name FROM yaml_dumps WHERE dump_type = ?;', ( dump_type, ) ) ]
+        
+        if dump_type == YAML_DUMP_ID_LOCAL_BOORU:
+            
+            names = [ bytes.fromhex( name ) for name in names ]
+            
+        
+        return names
+        
+    
+    def OverwriteJSONDumps( self, dump_types, objs ):
+        
+        for dump_type in dump_types:
+            
+            self.DeleteJSONDumpNamed( dump_type )
+            
+        
+        for obj in objs:
+            
+            self.SetJSONDump( obj )
+            
+        
+    
+    def SetJSONDump( self, obj ):
+        
+        if isinstance( obj, HydrusSerialisable.SerialisableBaseNamed ):
+            
+            ( dump_type, dump_name, version, serialisable_info ) = obj.GetSerialisableTuple()
+            
+            try:
+                
+                dump = json.dumps( serialisable_info )
+                
+            except Exception as e:
+                
+                HydrusData.ShowException( e )
+                HydrusData.Print( obj )
+                HydrusData.Print( serialisable_info )
+                
+                raise Exception( 'Trying to json dump the object ' + str( obj ) + ' with name ' + dump_name + ' caused an error. Its serialisable info has been dumped to the log.' )
+                
+            
+            store_backups = False
+            
+            if dump_type == HydrusSerialisable.SERIALISABLE_TYPE_GUI_SESSION:
+                
+                store_backups = True
+                backup_depth = HG.client_controller.new_options.GetInteger( 'number_of_gui_session_backups' )
+                
+            
+            object_timestamp = HydrusData.GetNow()
+            
+            if store_backups:
+                
+                existing_timestamps = sorted( self._STI( self._c.execute( 'SELECT timestamp FROM json_dumps_named WHERE dump_type = ? AND dump_name = ?;', ( dump_type, dump_name ) ) ) )
+                
+                if len( existing_timestamps ) > 0:
+                    
+                    # the user has changed their system clock, so let's make sure the new timestamp is larger at least
+                    
+                    largest_existing_timestamp = max( existing_timestamps )
+                    
+                    if largest_existing_timestamp > object_timestamp:
+                        
+                        object_timestamp = largest_existing_timestamp + 1
+                        
+                    
+                
+                deletee_timestamps = existing_timestamps[ : - backup_depth ] # keep highest n values
+                
+                deletee_timestamps.append( object_timestamp ) # if save gets spammed twice in one second, we'll overwrite
+                
+                self._c.executemany( 'DELETE FROM json_dumps_named WHERE dump_type = ? AND dump_name = ? AND timestamp = ?;', [ ( dump_type, dump_name, timestamp ) for timestamp in deletee_timestamps ] )
+                
+            else:
+                
+                self._c.execute( 'DELETE FROM json_dumps_named WHERE dump_type = ? AND dump_name = ?;', ( dump_type, dump_name ) )
+                
+            
+            dump_buffer = GenerateBigSQLiteDumpBuffer( dump )
+            
+            try:
+                
+                self._c.execute( 'INSERT INTO json_dumps_named ( dump_type, dump_name, version, timestamp, dump ) VALUES ( ?, ?, ?, ?, ? );', ( dump_type, dump_name, version, object_timestamp, dump_buffer ) )
+                
+            except:
+                
+                HydrusData.DebugPrint( dump )
+                HydrusData.ShowText( 'Had a problem saving a JSON object. The dump has been printed to the log.' )
+                
+                raise
+                
+            
+        else:
+            
+            ( dump_type, version, serialisable_info ) = obj.GetSerialisableTuple()
+            
+            if dump_type == HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER:
+                
+                deletee_session_names = obj.GetDeleteeSessionNames()
+                dirty_session_containers = obj.GetDirtySessionContainers()
+                
+                if len( deletee_session_names ) > 0:
+                    
+                    for deletee_session_name in deletee_session_names:
+                        
+                        self.DeleteJSONDumpNamed( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER_SESSION_CONTAINER, dump_name = deletee_session_name )
+                        
+                    
+                
+                if len( dirty_session_containers ) > 0:
+                    
+                    for dirty_session_container in dirty_session_containers:
+                        
+                        self.SetJSONDump( dirty_session_container )
+                        
+                    
+                
+                if not obj.IsDirty():
+                    
+                    return
+                    
+                
+            elif dump_type == HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_BANDWIDTH_MANAGER:
+                
+                deletee_tracker_names = obj.GetDeleteeTrackerNames()
+                dirty_tracker_containers = obj.GetDirtyTrackerContainers()
+                
+                if len( deletee_tracker_names ) > 0:
+                    
+                    for deletee_tracker_name in deletee_tracker_names:
+                        
+                        self.DeleteJSONDumpNamed( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_BANDWIDTH_MANAGER_TRACKER_CONTAINER, dump_name = deletee_tracker_name )
+                        
+                    
+                
+                if len( dirty_tracker_containers ) > 0:
+                    
+                    for dirty_tracker_container in dirty_tracker_containers:
+                        
+                        self.SetJSONDump( dirty_tracker_container )
+                        
+                    
+                
+                if not obj.IsDirty():
+                    
+                    return
+                    
+                
+            
+            try:
+                
+                dump = json.dumps( serialisable_info )
+                
+            except Exception as e:
+                
+                HydrusData.ShowException( e )
+                HydrusData.Print( obj )
+                HydrusData.Print( serialisable_info )
+                
+                raise Exception( 'Trying to json dump the object ' + str( obj ) + ' caused an error. Its serialisable info has been dumped to the log.' )
+                
+            
+            self._c.execute( 'DELETE FROM json_dumps WHERE dump_type = ?;', ( dump_type, ) )
+            
+            dump_buffer = GenerateBigSQLiteDumpBuffer( dump )
+            
+            try:
+                
+                self._c.execute( 'INSERT INTO json_dumps ( dump_type, version, dump ) VALUES ( ?, ?, ? );', ( dump_type, version, dump_buffer ) )
+                
+            except:
+                
+                HydrusData.DebugPrint( dump )
+                HydrusData.ShowText( 'Had a problem saving a JSON object. The dump has been printed to the log.' )
+                
+                raise
+                
+            
+        
+    
+    def SetJSONComplex( self,
+        overwrite_types_and_objs: typing.Optional[ typing.Tuple[ typing.Iterable[ int ], typing.Iterable[ HydrusSerialisable.SerialisableBase ] ] ] = None,
+        set_objs: typing.Optional[ typing.List[ HydrusSerialisable.SerialisableBase ] ] = None,
+        deletee_types_to_names: typing.Optional[ typing.Dict[ int, typing.Iterable[ str ] ] ] = None
+    ):
+        
+        if overwrite_types_and_objs is not None:
+            
+            ( dump_types, objs ) = overwrite_types_and_objs
+            
+            self.OverwriteJSONDumps( dump_types, objs )
+            
+        
+        if set_objs is not None:
+            
+            for obj in set_objs:
+                
+                self.SetJSONDump( obj )
+                
+            
+        
+        if deletee_types_to_names is not None:
+            
+            for ( dump_type, names ) in deletee_types_to_names.items():
+                
+                for name in names:
+                    
+                    self.DeleteJSONDumpNamed( dump_type, dump_name = name )
+                    
+                
+            
+        
+    
+    def SetJSONSimple( self, name, value ):
+        
+        if value is None:
+            
+            self._c.execute( 'DELETE FROM json_dict WHERE name = ?;', ( name, ) )
+            
+        else:
+            
+            dump = json.dumps( value )
+            
+            dump_buffer = GenerateBigSQLiteDumpBuffer( dump )
+            
+            try:
+                
+                self._c.execute( 'REPLACE INTO json_dict ( name, dump ) VALUES ( ?, ? );', ( name, dump_buffer ) )
+                
+            except:
+                
+                HydrusData.DebugPrint( dump )
+                HydrusData.ShowText( 'Had a problem saving a JSON object. The dump has been printed to the log.' )
+                
+                raise
+                
+            
+        
+    
+    def SetYAMLDump( self, dump_type, dump_name, data ):
+        
+        if dump_type == YAML_DUMP_ID_LOCAL_BOORU:
+            
+            dump_name = dump_name.hex()
+            
+        
+        self._c.execute( 'DELETE FROM yaml_dumps WHERE dump_type = ? AND dump_name = ?;', ( dump_type, dump_name ) )
+        
+        try:
+            
+            self._c.execute( 'INSERT INTO yaml_dumps ( dump_type, dump_name, dump ) VALUES ( ?, ?, ? );', ( dump_type, dump_name, data ) )
+            
+        except:
+            
+            HydrusData.Print( ( dump_type, dump_name, data ) )
+            
+            raise
+            
+        
+        if dump_type == YAML_DUMP_ID_LOCAL_BOORU:
+            
+            service_id = self.modules_services.GetServiceId( CC.LOCAL_BOORU_SERVICE_KEY )
+            
+            self._c.execute( 'DELETE FROM service_info WHERE service_id = ? AND info_type = ?;', ( service_id, HC.SERVICE_INFO_NUM_SHARES ) )
+            
+            HG.client_controller.pub( 'refresh_local_booru_shares' )
+            
+        
+    

hydrus/client/gui/ClientGUI.py

@@ -3194,6 +3194,31 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ):
             
         
     
+    def _RegenerateTagCacheSearchableSubtagsMaps( self ):
+        
+        message = 'This will regenerate the fast search cache\'s \'unusual character logic\' lookup map, for one or all tag services.'
+        message += os.linesep * 2
+        message += 'If you have a lot of tags, it can take a little while, during which the gui may hang.'
+        message += os.linesep * 2
+        message += 'If you do not have a specific reason to run this, it is pointless. It fixes missing autocomplete search results.'
+        
+        result = ClientGUIDialogsQuick.GetYesNo( self, message, yes_label = 'do it--now choose which service', no_label = 'forget it' )
+        
+        if result == QW.QDialog.Accepted:
+            
+            try:
+                
+                tag_service_key = GetTagServiceKeyForMaintenance( self )
+                
+            except HydrusExceptions.CancelledException:
+                
+                return
+                
+            
+            self._controller.Write( 'regenerate_searchable_subtag_maps', tag_service_key = tag_service_key )
+            
+        
+    
     def _RegenerateTagParentsLookupCache( self ):
         
         message = 'This will delete and then recreate the tag parents lookup cache, which is used for all basic tag parents operations. This is useful if it has become damaged or otherwise desynchronised.'
@@ -5012,6 +5037,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
             ClientGUIMenus.AppendMenuItem( submenu, 'tag parents lookup cache', 'Delete and recreate the tag siblings cache.', self._RegenerateTagParentsLookupCache )
             ClientGUIMenus.AppendMenuItem( submenu, 'tag text search cache', 'Delete and regenerate the cache hydrus uses for fast tag search.', self._RegenerateTagCache )
             ClientGUIMenus.AppendMenuItem( submenu, 'tag text search cache (subtags repopulation)', 'Repopulate the subtags for the cache hydrus uses for fast tag search.', self._RepopulateTagCacheMissingSubtags )
+            ClientGUIMenus.AppendMenuItem( submenu, 'tag text search cache (searchable subtag maps)', 'Regenerate the searchable subtag maps.', self._RegenerateTagCacheSearchableSubtagsMaps )
             
             ClientGUIMenus.AppendSeparator( submenu )
             

hydrus/client/gui/ClientGUIApplicationCommand.py

@@ -402,9 +402,7 @@ class TagSubPanel( QW.QWidget ):
         self._tag_value = QW.QLineEdit( self )
         self._tag_value.setReadOnly( True )
         
-        expand_parents = False
-        
-        self._tag_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.SetTags, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, CC.COMBINED_TAG_SERVICE_KEY )
+        self._tag_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.SetTags, CC.LOCAL_FILE_SERVICE_KEY, CC.COMBINED_TAG_SERVICE_KEY )
         
         #
         
@@ -418,6 +416,8 @@ class TagSubPanel( QW.QWidget ):
             self._service_keys.addItem( service_name, service_key )
             
         
+        self._tag_input.SetTagServiceKey( self._service_keys.GetValue() )
+        
         #
         
         vbox = QP.VBoxLayout()
@@ -437,6 +437,15 @@ class TagSubPanel( QW.QWidget ):
         
         self.setLayout( vbox )
         
+        #
+        
+        self._service_keys.currentIndexChanged.connect( self._NewServiceKey )
+        
+    
+    def _NewServiceKey( self ):
+        
+        self._tag_input.SetTagServiceKey( self._service_keys.GetValue() )
+        
     
     def GetValue( self ):
         

hydrus/client/gui/ClientGUICanvas.py

@@ -40,6 +40,7 @@ from hydrus.client.gui import QtPorting as QP
 from hydrus.client.media import ClientMedia
 from hydrus.client.metadata import ClientRatings
 from hydrus.client.metadata import ClientTags
+from hydrus.client.metadata import ClientTagSorting
 
 ZOOM_CENTERPOINT_MEDIA_CENTER = 0
 ZOOM_CENTERPOINT_VIEWER_CENTER = 1
@@ -2000,7 +2001,7 @@ class CanvasWithDetails( Canvas ):
             
             tags_i_want_to_display = list( tags_i_want_to_display )
             
-            ClientTags.SortTags( HC.options[ 'default_tag_sort' ], tags_i_want_to_display )
+            ClientTagSorting.SortTags( HC.options[ 'default_tag_sort' ], tags_i_want_to_display )
             
             current_y = 3
             

hydrus/client/gui/ClientGUICommon.py

@@ -490,6 +490,13 @@ class BetterStaticText( QP.EllipsizedLabel ):
             
         
     
+    def clear( self ):
+        
+        self._last_set_text = ''
+        
+        QP.EllipsizedLabel.clear( self )
+        
+    
     def setText( self, text ):
         
         # this doesn't need mnemonic escape _unless_ a buddy is set, wew lad

hydrus/client/gui/ClientGUIControls.py

@@ -21,6 +21,7 @@ from hydrus.client.gui import ClientGUITopLevelWindowsPanels
 from hydrus.client.gui import QtPorting as QP
 from hydrus.client.gui.lists import ClientGUIListConstants as CGLC
 from hydrus.client.gui.lists import ClientGUIListCtrl
+from hydrus.client.networking import ClientNetworkingJobs
 
 class BandwidthRulesCtrl( ClientGUICommon.StaticBox ):
     
@@ -297,7 +298,7 @@ class BytesControl( QW.QWidget ):
     def _HandleValueChanged( self, val ):
         
         self.valueChanged.emit()
-              
+        
     
     def GetSeparatedValue( self ):
         
@@ -430,7 +431,6 @@ class NetworkJobControl( QW.QFrame ):
         self.setFrameStyle( QW.QFrame.Box | QW.QFrame.Raised )
         
         self._network_job = None
-        self._download_started = False
         
         self._auto_override_bandwidth_rules = False
         
@@ -545,25 +545,17 @@ class NetworkJobControl( QW.QFrame ):
             
             self._left_text.setText( status_text )
             
-            if not self._download_started and current_speed > 0:
-                
-                self._download_started = True
-                
-            
             speed_text = ''
             
-            if self._download_started and not self._network_job.HasError():
+            if bytes_read is not None and bytes_read > 0 and not self._network_job.HasError():
                 
-                if bytes_read is not None:
+                if bytes_to_read is not None and bytes_read != bytes_to_read:
                     
-                    if bytes_to_read is not None and bytes_read != bytes_to_read:
-                        
-                        speed_text += HydrusData.ConvertValueRangeToBytes( bytes_read, bytes_to_read )
-                        
-                    else:
-                        
-                        speed_text += HydrusData.ToHumanBytes( bytes_read )
-                        
+                    speed_text += HydrusData.ConvertValueRangeToBytes( bytes_read, bytes_to_read )
+                    
+                else:
+                    
+                    speed_text += HydrusData.ToHumanBytes( bytes_read )
                     
                 
                 if current_speed != bytes_to_read: # if it is a real quick download, just say its size
@@ -623,7 +615,7 @@ class NetworkJobControl( QW.QFrame ):
         self._auto_override_bandwidth_rules = not self._auto_override_bandwidth_rules
         
     
-    def SetNetworkJob( self, network_job ):
+    def SetNetworkJob( self, network_job: typing.Optional[ ClientNetworkingJobs.NetworkJob ] ):
         
         if network_job is None:
             
@@ -641,7 +633,8 @@ class NetworkJobControl( QW.QFrame ):
             if self._network_job != network_job:
                 
                 self._network_job = network_job
-                self._download_started = False
+                
+                self._Update()
                 
                 HG.client_controller.gui.RegisterUIUpdateWindow( self )
                 

hydrus/client/gui/ClientGUIDialogs.py

@@ -506,17 +506,15 @@ class DialogInputNamespaceRegex( Dialog ):
     
 class DialogInputTags( Dialog ):
     
-    def __init__( self, parent, service_key, tags, expand_parents = True, message = '', show_display_decorators = False ):
+    def __init__( self, parent, service_key, tag_display_type, tags, message = '' ):
         
         Dialog.__init__( self, parent, 'input tags' )
         
         self._service_key = service_key
         
-        self._tags = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self, service_key = service_key, show_display_decorators = show_display_decorators )
+        self._tags = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self, service_key, tag_display_type )
         
-        self._expand_parents = expand_parents
-        
-        self._tag_autocomplete = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterTags, self._expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key, null_entry_callable = self.OK, show_paste_button = True )
+        self._tag_autocomplete = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterTags, CC.LOCAL_FILE_SERVICE_KEY, service_key, null_entry_callable = self.OK, show_paste_button = True )
         
         self._ok = ClientGUICommon.BetterButton( self, 'OK', self.done, QW.QDialog.Accepted )
         self._ok.setObjectName( 'HydrusAccept' )

hydrus/client/gui/ClientGUIImport.py

@@ -485,11 +485,9 @@ class FilenameTaggingOptionsPanel( QW.QWidget ):
             
             self._tags_panel = ClientGUICommon.StaticBox( self, 'tags for all' )
             
-            self._tags = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self._tags_panel, self._service_key, self.TagsRemoved )
+            self._tags = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self._tags_panel, self._service_key, ClientTags.TAG_DISPLAY_STORAGE, self.TagsRemoved )
             
-            expand_parents = True
-            
-            self._tag_autocomplete_all = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self._tags_panel, self.EnterTags, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
+            self._tag_autocomplete_all = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self._tags_panel, self.EnterTags, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
             
             self._tags_paste_button = ClientGUICommon.BetterButton( self._tags_panel, 'paste tags', self._PasteTags )
             
@@ -499,13 +497,11 @@ class FilenameTaggingOptionsPanel( QW.QWidget ):
             
             self._paths_to_single_tags = collections.defaultdict( set )
             
-            self._single_tags = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self._single_tags_panel, self._service_key, self.SingleTagsRemoved )
+            self._single_tags = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self._single_tags_panel, self._service_key, ClientTags.TAG_DISPLAY_STORAGE, self.SingleTagsRemoved )
             
             self._single_tags_paste_button = ClientGUICommon.BetterButton( self._single_tags_panel, 'paste tags', self._PasteSingleTags )
             
-            expand_parents = True
-            
-            self._tag_autocomplete_selection = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self._single_tags_panel, self.EnterTagsSingle, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
+            self._tag_autocomplete_selection = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self._single_tags_panel, self.EnterTagsSingle, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
             
             self.SetSelectedPaths( [] )
             

hydrus/client/gui/ClientGUIManagement.py

@@ -667,7 +667,7 @@ HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIAL
 
 class ListBoxTagsMediaManagementPanel( ClientGUIListBoxes.ListBoxTagsMedia ):
     
-    def __init__( self, parent, management_controller: ManagementController, page_key, tag_display_type, tag_autocomplete: typing.Optional[ ClientGUIACDropdown.AutoCompleteDropdownTagsRead ] = None ):
+    def __init__( self, parent, management_controller: ManagementController, page_key, tag_display_type = ClientTags.TAG_DISPLAY_SELECTION_LIST, tag_autocomplete: typing.Optional[ ClientGUIACDropdown.AutoCompleteDropdownTagsRead ] = None ):
         
         ClientGUIListBoxes.ListBoxTagsMedia.__init__( self, parent, tag_display_type, include_counts = True )
         
@@ -766,7 +766,6 @@ def managementScrollbarValueChanged( value ):
 class ManagementPanel( QW.QScrollArea ):
     
     SHOW_COLLECT = True
-    TAG_DISPLAY_TYPE = ClientTags.TAG_DISPLAY_SELECTION_LIST
     
     def __init__( self, parent, page, controller, management_controller ):
         
@@ -840,7 +839,7 @@ class ManagementPanel( QW.QScrollArea ):
         
         tags_box = ClientGUIListBoxes.StaticBoxSorterForListBoxTags( self, 'selection tags' )
         
-        self._current_selection_tags_list = ListBoxTagsMediaManagementPanel( tags_box, self._management_controller, self._page_key, self.TAG_DISPLAY_TYPE )
+        self._current_selection_tags_list = ListBoxTagsMediaManagementPanel( tags_box, self._management_controller, self._page_key )
         
         tags_box.SetTagsBox( self._current_selection_tags_list )
         
@@ -4668,7 +4667,7 @@ class ManagementPanelQuery( ManagementPanel ):
         
         if self._search_enabled:
             
-            self._current_selection_tags_list = ListBoxTagsMediaManagementPanel( tags_box, self._management_controller, self._page_key, self.TAG_DISPLAY_TYPE, tag_autocomplete = self._tag_autocomplete )
+            self._current_selection_tags_list = ListBoxTagsMediaManagementPanel( tags_box, self._management_controller, self._page_key, tag_autocomplete = self._tag_autocomplete )
             
             file_search_context = self._management_controller.GetVariable( 'file_search_context' )
             
@@ -4682,7 +4681,7 @@ class ManagementPanelQuery( ManagementPanel ):
             
         else:
             
-            self._current_selection_tags_list = ListBoxTagsMediaManagementPanel( tags_box, self._management_controller, self._page_key, self.TAG_DISPLAY_TYPE )
+            self._current_selection_tags_list = ListBoxTagsMediaManagementPanel( tags_box, self._management_controller, self._page_key )
             
         
         tags_box.SetTagsBox( self._current_selection_tags_list )

hydrus/client/gui/ClientGUIResults.py

@@ -3514,12 +3514,12 @@ class MediaPanelThumbnails( MediaPanel ):
             
             if len( disparate_petitioned_file_service_keys ) > 0:
                 
-                ClientGUIMedia.AddServiceKeyLabelsToMenu( selection_info_menu, disparate_petitioned_file_service_keys, 'some petitioned from' )
+                ClientGUIMedia.AddServiceKeyLabelsToMenu( selection_info_menu, disparate_petitioned_file_service_keys, 'some petitioned for removal from' )
                 
             
             if len( common_petitioned_file_service_keys ) > 0:
                 
-                ClientGUIMedia.AddServiceKeyLabelsToMenu( selection_info_menu, common_petitioned_file_service_keys, 'petitioned from' )
+                ClientGUIMedia.AddServiceKeyLabelsToMenu( selection_info_menu, common_petitioned_file_service_keys, 'petitioned for removal from' )
                 
             
             if len( disparate_deleted_file_service_keys ) > 0:

hydrus/client/gui/ClientGUIScrolledPanelsEdit.py

@@ -2294,7 +2294,7 @@ class EditTagImportOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
         message += os.linesep * 2
         message += 'This is usually easier and faster to do just by adding tags to the downloader query (e.g. "artistname desired_tag"), so reserve this for downloaders that do not work on tags or where you want to whitelist multiple tags.'
         
-        with ClientGUIDialogs.DialogInputTags( self, CC.COMBINED_TAG_SERVICE_KEY, list( self._tag_whitelist ), expand_parents = False, message = message ) as dlg:
+        with ClientGUIDialogs.DialogInputTags( self, CC.COMBINED_TAG_SERVICE_KEY, ClientTags.TAG_DISPLAY_ACTUAL, list( self._tag_whitelist ), message = message ) as dlg:
             
             if dlg.exec() == QW.QDialog.Accepted:
                 
@@ -2690,7 +2690,7 @@ class EditServiceTagImportOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
         
         message = 'Any tags you enter here will be applied to every file that passes through this import context.'
         
-        with ClientGUIDialogs.DialogInputTags( self, self._service_key, list( self._additional_tags ), message = message, show_display_decorators = True ) as dlg:
+        with ClientGUIDialogs.DialogInputTags( self, self._service_key, ClientTags.TAG_DISPLAY_STORAGE, list( self._additional_tags ), message = message ) as dlg:
             
             if dlg.exec() == QW.QDialog.Accepted:
                 

hydrus/client/gui/ClientGUIScrolledPanelsManagement.py

@@ -39,6 +39,7 @@ from hydrus.client.gui.lists import ClientGUIListCtrl
 from hydrus.client.gui.search import ClientGUIACDropdown
 from hydrus.client.gui.search import ClientGUISearch
 from hydrus.client.media import ClientMedia
+from hydrus.client.metadata import ClientTags
 from hydrus.client.networking import ClientNetworkingSessions
 
 class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
@@ -2930,6 +2931,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             self._default_tag_service_search_page = ClientGUICommon.BetterChoice( general_panel )
             
+            self._expand_parents_on_storage_taglists = QW.QCheckBox( general_panel )
+            self._expand_parents_on_storage_autocomplete_taglists = QW.QCheckBox( general_panel )
             self._ac_select_first_with_count = QW.QCheckBox( general_panel )
             
             #
@@ -2941,10 +2944,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             favourites_st = ClientGUICommon.BetterStaticText( favourites_panel, desc )
             favourites_st.setWordWrap( True )
             
-            expand_parents = False
-            
-            self._favourites = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( favourites_panel, show_display_decorators = False )
-            self._favourites_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( favourites_panel, self._favourites.AddTags, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, CC.COMBINED_TAG_SERVICE_KEY, show_paste_button = True )
+            self._favourites = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( favourites_panel, CC.COMBINED_TAG_SERVICE_KEY, ClientTags.TAG_DISPLAY_STORAGE )
+            self._favourites_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( favourites_panel, self._favourites.AddTags, CC.LOCAL_FILE_SERVICE_KEY, CC.COMBINED_TAG_SERVICE_KEY, show_paste_button = True )
             
             #
             
@@ -2967,6 +2968,14 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             self._default_tag_service_search_page.SetValue( new_options.GetKey( 'default_tag_service_search_page' ) )
             
+            self._expand_parents_on_storage_taglists.setChecked( self._new_options.GetBoolean( 'expand_parents_on_storage_taglists' ) )
+            
+            self._expand_parents_on_storage_taglists.setToolTip( 'This affects taglists in places like the manage tags dialog, where you edit tags as they actually are, and implied parents hang below tags.' )
+            
+            self._expand_parents_on_storage_autocomplete_taglists.setChecked( self._new_options.GetBoolean( 'expand_parents_on_storage_autocomplete_taglists' ) )
+            
+            self._expand_parents_on_storage_autocomplete_taglists.setToolTip( 'This affects the autocomplete results taglist.' )
+            
             self._ac_select_first_with_count.setChecked( self._new_options.GetBoolean( 'ac_select_first_with_count' ) )
             
             #
@@ -2982,6 +2991,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             rows.append( ( 'Default tag service in manage tag dialogs: ', self._default_tag_repository ) )
             rows.append( ( 'Default tag service in search pages: ', self._default_tag_service_search_page ) )
             rows.append( ( 'Default tag sort: ', self._default_tag_sort ) )
+            rows.append( ( 'Show parents expanded by default on edit/write taglists: ', self._expand_parents_on_storage_taglists ) )
+            rows.append( ( 'Show parents expanded by default on edit/write autocomplete taglists: ', self._expand_parents_on_storage_autocomplete_taglists ) )
             rows.append( ( 'By default, select the first tag result with actual count in write-autocomplete: ', self._ac_select_first_with_count ) )
             
             gridbox = ClientGUICommon.WrapInGrid( general_panel, rows )
@@ -3008,6 +3019,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             HC.options[ 'default_tag_repository' ] = self._default_tag_repository.GetValue()
             HC.options[ 'default_tag_sort' ] = QP.GetClientData( self._default_tag_sort, self._default_tag_sort.currentIndex() )
             
+            self._new_options.SetBoolean( 'expand_parents_on_storage_taglists', self._expand_parents_on_storage_taglists.isChecked() )
+            self._new_options.SetBoolean( 'expand_parents_on_storage_autocomplete_taglists', self._expand_parents_on_storage_autocomplete_taglists.isChecked() )
             self._new_options.SetBoolean( 'ac_select_first_with_count', self._ac_select_first_with_count.isChecked() )
             
             self._new_options.SetKey( 'default_tag_service_search_page', self._default_tag_service_search_page.GetValue() )
@@ -3187,15 +3200,13 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
                 self._suggested_favourites_services.addItem( tag_service.GetName(), tag_service.GetServiceKey() )
                 
             
-            self._suggested_favourites = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( suggested_tags_favourites_panel )
+            self._suggested_favourites = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( suggested_tags_favourites_panel, CC.COMBINED_TAG_SERVICE_KEY, ClientTags.TAG_DISPLAY_STORAGE )
             
             self._current_suggested_favourites_service = None
             
             self._suggested_favourites_dict = {}
             
-            expand_parents = False
-            
-            self._suggested_favourites_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( suggested_tags_favourites_panel, self._suggested_favourites.AddTags, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, CC.COMBINED_TAG_SERVICE_KEY, show_paste_button = True )
+            self._suggested_favourites_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( suggested_tags_favourites_panel, self._suggested_favourites.AddTags, CC.LOCAL_FILE_SERVICE_KEY, CC.COMBINED_TAG_SERVICE_KEY, show_paste_button = True )
             
             #
             
@@ -3365,7 +3376,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             self._suggested_favourites.SetTags( favourites )
             
-            self._suggested_favourites_input.SetTagService( self._current_suggested_favourites_service )
+            self._suggested_favourites_input.SetTagServiceKey( self._current_suggested_favourites_service )
             self._suggested_favourites_input.SetDisplayTagServiceKey( self._current_suggested_favourites_service )
             
         

hydrus/client/gui/ClientGUIScrolledPanelsReview.py

@@ -1187,7 +1187,7 @@ class MigrateTagsPanel( ClientGUIScrolledPanels.ReviewPanel ):
         destination_action_strings[ HC.CONTENT_UPDATE_DELETE ] = 'deleting them from'
         destination_action_strings[ HC.CONTENT_UPDATE_CLEAR_DELETE_RECORD ] = 'clearing their deletion record from'
         destination_action_strings[ HC.CONTENT_UPDATE_PEND ] = 'pending them to'
-        destination_action_strings[ HC.CONTENT_UPDATE_PETITION ] = 'petitioning them from'
+        destination_action_strings[ HC.CONTENT_UPDATE_PETITION ] = 'petitioning them for removal from'
         
         content_type = self._migration_content_type.GetValue()
         content_statuses = self._migration_source_content_status_filter.GetValue()

hydrus/client/gui/ClientGUITagSuggestions.py

@@ -21,6 +21,7 @@ from hydrus.client.gui import QtPorting as QP
 from hydrus.client.gui.lists import ClientGUIListBoxes
 from hydrus.client.gui.lists import ClientGUIListBoxesData
 from hydrus.client.metadata import ClientTags
+from hydrus.client.metadata import ClientTagSorting
 
 def FilterSuggestedPredicatesForMedia( predicates: typing.Sequence[ ClientSearch.Predicate ], medias: typing.Collection[ ClientMedia.Media ], service_key: bytes ) -> typing.List[ ClientSearch.Predicate ]:
     
@@ -58,7 +59,7 @@ class ListBoxTagsSuggestionsFavourites( ClientGUIListBoxes.ListBoxTagsStrings ):
     
     def __init__( self, parent, service_key, activate_callable, sort_tags = True ):
         
-        ClientGUIListBoxes.ListBoxTagsStrings.__init__( self, parent, service_key = service_key, sort_tags = sort_tags, render_for_user = False )
+        ClientGUIListBoxes.ListBoxTagsStrings.__init__( self, parent, service_key = service_key, sort_tags = sort_tags, tag_display_type = ClientTags.TAG_DISPLAY_STORAGE )
         
         self._activate_callable = activate_callable
         
@@ -104,7 +105,7 @@ class ListBoxTagsSuggestionsRelated( ClientGUIListBoxes.ListBoxTagsPredicates ):
     
     def __init__( self, parent, service_key, activate_callable ):
         
-        ClientGUIListBoxes.ListBoxTagsPredicates.__init__( self, parent, render_for_user = False )
+        ClientGUIListBoxes.ListBoxTagsPredicates.__init__( self, parent, tag_display_type = ClientTags.TAG_DISPLAY_STORAGE )
         
         self._activate_callable = activate_callable
         
@@ -131,13 +132,11 @@ class ListBoxTagsSuggestionsRelated( ClientGUIListBoxes.ListBoxTagsPredicates ):
         return False
         
     
-    def _GenerateTermFromPredicate( self, predicate: ClientSearch.Predicate ):
+    def _GenerateTermFromPredicate( self, predicate: ClientSearch.Predicate ) -> ClientGUIListBoxesData.ListBoxItemPredicate:
         
         predicate.ClearCounts()
         
-        show_ideal_siblings = True
-        
-        return ClientGUIListBoxesData.ListBoxItemPredicate( predicate, show_ideal_siblings )
+        return ClientGUIListBoxesData.ListBoxItemPredicate( predicate )
         
     
     def TakeFocusForUser( self ):
@@ -175,7 +174,7 @@ class FavouritesTagsPanel( QW.QWidget ):
         
         favourites = list( HG.client_controller.new_options.GetSuggestedTagsFavourites( self._service_key ) )
         
-        ClientTags.SortTags( HC.options[ 'default_tag_sort' ], favourites )
+        ClientTagSorting.SortTags( HC.options[ 'default_tag_sort' ], favourites )
         
         tags = FilterSuggestedTagsForMedia( favourites, self._media, self._service_key )
         

hydrus/client/gui/ClientGUITags.py

@@ -2021,7 +2021,7 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
                 
             else:
                 
-                text = 'petition all/selected tags'
+                text = 'petition to remove all/selected tags'
                 
             
             self._remove_tags = ClientGUICommon.BetterButton( self._tags_box_sorter, text, self._RemoveTagsButton )
@@ -2063,9 +2063,7 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             #
             
-            expand_parents = True
-            
-            self._add_tag_box = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.AddTags, expand_parents, self._file_service_key, self._tag_service_key, null_entry_callable = self.OK )
+            self._add_tag_box = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.AddTags, self._file_service_key, self._tag_service_key, null_entry_callable = self.OK )
             
             self._tags_box.SetTagServiceKey( self._tag_service_key )
             
@@ -2202,7 +2200,7 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             if len( choices ) == 1:
                 
-                [ ( choice_action, tag_counts ) ] = list(choices.items())
+                [ ( choice_action, tag_counts ) ] = list( choices.items() )
                 
                 tags = { tag for ( tag, count ) in tag_counts }
                 
@@ -2216,10 +2214,19 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
                 
                 choice_text_lookup[ HC.CONTENT_UPDATE_ADD ] = 'add'
                 choice_text_lookup[ HC.CONTENT_UPDATE_DELETE ] = 'delete'
-                choice_text_lookup[ HC.CONTENT_UPDATE_PEND ] = 'pend'
-                choice_text_lookup[ HC.CONTENT_UPDATE_PETITION ] = 'petition'
-                choice_text_lookup[ HC.CONTENT_UPDATE_RESCIND_PEND ] = 'rescind pend'
-                choice_text_lookup[ HC.CONTENT_UPDATE_RESCIND_PETITION ] = 'rescind petition'
+                choice_text_lookup[ HC.CONTENT_UPDATE_PEND ] = 'pend (add)'
+                choice_text_lookup[ HC.CONTENT_UPDATE_PETITION ] = 'petition to remove'
+                choice_text_lookup[ HC.CONTENT_UPDATE_RESCIND_PEND ] = 'undo pend'
+                choice_text_lookup[ HC.CONTENT_UPDATE_RESCIND_PETITION ] = 'undo petition to remove'
+                
+                choice_tooltip_lookup = {}
+                
+                choice_tooltip_lookup[ HC.CONTENT_UPDATE_ADD ] = 'this adds the tags to this local tag service'
+                choice_tooltip_lookup[ HC.CONTENT_UPDATE_DELETE ] = 'this deletes the tags from this local tag service'
+                choice_tooltip_lookup[ HC.CONTENT_UPDATE_PEND ] = 'this pends the tags to be added to this tag repository when you upload'
+                choice_tooltip_lookup[ HC.CONTENT_UPDATE_PETITION ] = 'this petitions the tags for deletion from this tag repository when you upload'
+                choice_tooltip_lookup[ HC.CONTENT_UPDATE_RESCIND_PEND ] = 'this rescinds the currently pending tags, so they will not be added'
+                choice_tooltip_lookup[ HC.CONTENT_UPDATE_RESCIND_PETITION ] = 'this rescinds the current tag petitions, so they will not be deleted'
                 
                 for choice_action in preferred_order:
                     
@@ -2232,42 +2239,56 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
                     
                     tag_counts = choices[ choice_action ]
                     
-                    tags = { tag for ( tag, count ) in tag_counts }
+                    choice_tags = { tag for ( tag, count ) in tag_counts }
                     
-                    if len( tags ) == 1:
+                    if len( choice_tags ) == 1:
                         
                         [ ( tag, count ) ] = tag_counts
                         
-                        text = choice_text_prefix + ' "' + HydrusText.ElideText( tag, 64 ) + '" for ' + HydrusData.ToHumanInt( count ) + ' files'
+                        text = '{} "{}" for {} files'.format( choice_text_prefix, HydrusText.ElideText( tag, 64 ), HydrusData.ToHumanInt( count ) )
                         
                     else:
                         
-                        text = choice_text_prefix + ' ' + HydrusData.ToHumanInt( len( tags ) ) + ' tags'
+                        text = '{} {} tags'.format( choice_text_prefix, HydrusData.ToHumanInt( len( choice_tags ) ) )
                         
                     
-                    data = ( choice_action, tags )
+                    data = ( choice_action, choice_tags )
+                    
+                    t_c_lines = [ choice_tooltip_lookup[ choice_action ] ]
                     
                     if len( tag_counts ) > 25:
                         
                         t_c = tag_counts[:25]
                         
-                        t_c_lines = [ tag + ' - ' + HydrusData.ToHumanInt( count ) + ' files' for ( tag, count ) in t_c ]
-                        
-                        t_c_lines.append( 'and ' + HydrusData.ToHumanInt( len( tag_counts ) - 25 ) + ' others' )
-                        
-                        tooltip = os.linesep.join( t_c_lines )
-                        
                     else:
                         
-                        tooltip = os.linesep.join( ( tag + ' - ' + HydrusData.ToHumanInt( count ) + ' files' for ( tag, count ) in tag_counts ) )
+                        t_c = tag_counts
                         
                     
+                    t_c_lines.extend( ( '{} - {} files'.format( tag, HydrusData.ToHumanInt( count ) ) for ( tag, count ) in t_c ) )
+                    
+                    if len( tag_counts ) > 25:
+                        
+                        t_c_lines.append( 'and {} others'.format( HydrusData.ToHumanInt( len( tag_counts ) - 25 ) ) )
+                        
+                    
+                    tooltip = os.linesep.join( t_c_lines )
+                    
                     bdc_choices.append( ( text, data, tooltip ) )
                     
                 
                 try:
                     
-                    ( choice_action, tags ) = ClientGUIDialogsQuick.SelectFromListButtons( self, 'What would you like to do?', bdc_choices )
+                    if len( tags ) > 1:
+                        
+                        message = 'The file{} some of those tags, but not all, so there are different things you can do.'.format( 's have' if len( self._media ) > 1 else ' has' )
+                        
+                    else:
+                        
+                        message = 'Of the {} files being managed, some that tag, but not all of them do, so there are different things you can do.'.format( HydrusData.ToHumanInt( len( self._media ) ) )
+                        
+                    
+                    ( choice_action, tags ) = ClientGUIDialogsQuick.SelectFromListButtons( self, 'What would you like to do?', bdc_choices, message = message )
                     
                 except HydrusExceptions.CancelledException:
                     
@@ -2822,20 +2843,18 @@ class ManageTagParents( ClientGUIScrolledPanels.ManagePanel ):
             
             listctrl_panel.AddMenuButton( 'export', menu_items, enabled_only_on_selection = True )
             
-            self._children = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self, self._service_key, show_display_decorators = False )
-            self._parents = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self, self._service_key, show_display_decorators = False )
+            self._children = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self, self._service_key, ClientTags.TAG_DISPLAY_ACTUAL )
+            self._parents = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self, self._service_key, ClientTags.TAG_DISPLAY_ACTUAL )
             
             ( gumpf, preview_height ) = ClientGUIFunctions.ConvertTextToPixels( self._children, ( 12, 6 ) )
             
             self._children.setMinimumHeight( preview_height )
             self._parents.setMinimumHeight( preview_height )
             
-            expand_parents = True
-            
-            self._child_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterChildren, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
+            self._child_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterChildren, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
             self._child_input.setEnabled( False )
             
-            self._parent_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterParents, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
+            self._parent_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterParents, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
             self._parent_input.setEnabled( False )
             
             self._add = QW.QPushButton( 'add', self )
@@ -2955,7 +2974,7 @@ class ManageTagParents( ClientGUIScrolledPanels.ManagePanel ):
                             pair_strings = os.linesep.join( ( child + '->' + parent for ( child, parent ) in new_pairs ) )
                             
                         
-                        message = 'Enter a reason for:' + os.linesep * 2 + pair_strings + os.linesep * 2 + 'To be added. A janitor will review your petition.'
+                        message = 'Enter a reason for:' + os.linesep * 2 + pair_strings + os.linesep * 2 + 'To be added. A janitor will review your request.'
                         
                         suggestions = []
                         
@@ -3014,7 +3033,7 @@ class ManageTagParents( ClientGUIScrolledPanels.ManagePanel ):
                             message = 'The pair ' + pair_strings + ' already exists.'
                             
                         
-                        result = ClientGUIDialogsQuick.GetYesNo( self, message, title = 'Choose what to do.', yes_label = 'petition it', no_label = 'do nothing' )
+                        result = ClientGUIDialogsQuick.GetYesNo( self, message, title = 'Choose what to do.', yes_label = 'petition to remove', no_label = 'do nothing' )
                         
                         if result == QW.QDialog.Accepted:
                             
@@ -3759,19 +3778,17 @@ class ManageTagSiblings( ClientGUIScrolledPanels.ManagePanel ):
             
             listctrl_panel.AddMenuButton( 'export', menu_items, enabled_only_on_selection = True )
             
-            self._old_siblings = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self, self._service_key, show_display_decorators = False )
+            self._old_siblings = ClientGUIListBoxes.ListBoxTagsStringsAddRemove( self, self._service_key, ClientTags.TAG_DISPLAY_ACTUAL )
             self._new_sibling = ClientGUICommon.BetterStaticText( self )
             
             ( gumpf, preview_height ) = ClientGUIFunctions.ConvertTextToPixels( self._old_siblings, ( 12, 6 ) )
             
             self._old_siblings.setMinimumHeight( preview_height )
             
-            expand_parents = False
-            
-            self._old_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterOlds, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
+            self._old_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterOlds, CC.LOCAL_FILE_SERVICE_KEY, service_key, show_paste_button = True )
             self._old_input.setEnabled( False )
             
-            self._new_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.SetNew, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key )
+            self._new_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.SetNew, CC.LOCAL_FILE_SERVICE_KEY, service_key )
             self._new_input.setEnabled( False )
             
             self._add = QW.QPushButton( 'add', self )

hydrus/client/gui/lists/ClientGUIListBoxes.py

@@ -29,6 +29,7 @@ from hydrus.client.gui.search import ClientGUISearch
 from hydrus.client.gui.lists import ClientGUIListBoxesData
 from hydrus.client.media import ClientMedia
 from hydrus.client.metadata import ClientTags
+from hydrus.client.metadata import ClientTagSorting
 
 class BetterQListWidget( QW.QListWidget ):
     
@@ -918,7 +919,7 @@ class ListBox( QW.QScrollArea ):
     
     TEXT_X_PADDING = 3
     
-    def __init__( self, parent, height_num_chars = 10, has_async_text_info = False, render_for_user = False ):
+    def __init__( self, parent: QW.QWidget, child_rows_allowed: bool, terms_may_have_child_rows: bool, height_num_chars = 10, has_async_text_info = False ):
         
         QW.QScrollArea.__init__( self, parent )
         self.setFrameStyle( QW.QFrame.Panel | QW.QFrame.Sunken )
@@ -947,7 +948,8 @@ class ListBox( QW.QScrollArea ):
         
         self._num_rows_per_page = 0
         
-        self._render_for_user = render_for_user
+        self._child_rows_allowed = child_rows_allowed
+        self._terms_may_have_child_rows = terms_may_have_child_rows
         
         #
         
@@ -1029,11 +1031,11 @@ class ListBox( QW.QScrollArea ):
         pass
         
     
-    def _ApplyAsyncInfoToTerm( self, term, info ):
+    def _ApplyAsyncInfoToTerm( self, term, info ) -> typing.Tuple[ bool, bool ]:
         
         # this guy comes with the lock
         
-        pass
+        return ( False, False )
         
     
     def _DeleteActivate( self ):
@@ -1067,7 +1069,7 @@ class ListBox( QW.QScrollArea ):
                 self._StartAsyncTextInfoLookup( term )
                 
             
-            self._total_positional_rows += term.GetRowCount()
+            self._total_positional_rows += term.GetRowCount( self._child_rows_allowed )
             
         
         if len( previously_selected_terms ) > 0:
@@ -1440,7 +1442,8 @@ class ListBox( QW.QScrollArea ):
         
         def publish_callable( terms_to_info ):
             
-            rows_changed = False
+            any_sort_info_changed = False
+            any_num_rows_changed = False
             
             with self._async_text_info_lock:
                 
@@ -1456,20 +1459,26 @@ class ListBox( QW.QScrollArea ):
                         term = self._positional_indices_to_terms[ self._terms_to_positional_indices[ term ] ]
                         
                     
-                    old_num_rows = term.GetRowCount()
+                    ( sort_info_changed, num_rows_changed ) = self._ApplyAsyncInfoToTerm( term, info )
                     
-                    self._ApplyAsyncInfoToTerm( term, info )
-                    
-                    new_num_rows = term.GetRowCount()
-                    
-                    if old_num_rows != new_num_rows:
+                    if sort_info_changed:
                         
-                        rows_changed = True
+                        any_sort_info_changed = True
+                        
+                    
+                    if num_rows_changed:
+                        
+                        any_num_rows_changed = True
                         
                     
                 
             
-            if rows_changed:
+            if any_sort_info_changed:
+                
+                self._Sort()
+                # this does regentermstoindices
+                
+            elif any_num_rows_changed:
                 
                 self._RegenTermsToIndices()
                 
@@ -1630,7 +1639,7 @@ class ListBox( QW.QScrollArea ):
             self._terms_to_positional_indices[ term ] = self._total_positional_rows
             self._positional_indices_to_terms[ self._total_positional_rows ] = term
             
-            self._total_positional_rows += term.GetRowCount()
+            self._total_positional_rows += term.GetRowCount( self._child_rows_allowed )
             
         
     
@@ -1971,6 +1980,20 @@ class ListBox( QW.QScrollArea ):
             
         
     
+    def SetChildRowsAllowed( self, value: bool ):
+        
+        if self._terms_may_have_child_rows and self._child_rows_allowed != value:
+            
+            self._child_rows_allowed = value
+            
+            self._RegenTermsToIndices()
+            
+            self._SetVirtualSize()
+            
+            self.widget().update()
+            
+        
+    
     def SetMinimumHeightNumChars( self, minimum_height_num_chars ):
         
         self._minimum_height_num_chars = minimum_height_num_chars
@@ -2002,11 +2025,18 @@ class ListBoxTags( ListBox ):
     
     can_spawn_new_windows = True
     
-    def __init__( self, *args, **kwargs ):
+    def __init__( self, parent, *args, tag_display_type: int = ClientTags.TAG_DISPLAY_STORAGE, **kwargs ):
         
-        ListBox.__init__( self, *args, **kwargs )
+        self._tag_display_type = tag_display_type
         
-        self._tag_display_type = ClientTags.TAG_DISPLAY_STORAGE
+        child_rows_allowed = HG.client_controller.new_options.GetBoolean( 'expand_parents_on_storage_taglists' )
+        terms_may_have_child_rows = self._tag_display_type == ClientTags.TAG_DISPLAY_STORAGE
+        
+        ListBox.__init__( self, parent, child_rows_allowed, terms_may_have_child_rows, *args, **kwargs )
+        
+        self._render_for_user = not self._tag_display_type == ClientTags.TAG_DISPLAY_STORAGE
+        
+        self._sibling_decoration_allowed = self._tag_display_type == ClientTags.TAG_DISPLAY_STORAGE
         
         self._page_key = None # placeholder. if a subclass sets this, it changes menu behaviour to allow 'select this tag' menu pubsubs
         
@@ -2067,11 +2097,6 @@ class ListBoxTags( ListBox ):
         return set()
         
     
-    def _GetFallbackServiceKey( self ):
-        
-        return CC.COMBINED_TAG_SERVICE_KEY
-        
-    
     def _GetNamespaceColours( self ):
         
         return HC.options[ 'namespace_colours' ]
@@ -2086,7 +2111,7 @@ class ListBoxTags( ListBox ):
         
         namespace_colours = self._GetNamespaceColours()
         
-        rows_of_texts_and_namespaces = term.GetRowsOfPresentationTextsWithNamespaces( self._render_for_user )
+        rows_of_texts_and_namespaces = term.GetRowsOfPresentationTextsWithNamespaces( self._render_for_user, self._sibling_decoration_allowed, self._child_rows_allowed )
         
         rows_of_texts_and_colours = []
         
@@ -2317,6 +2342,34 @@ class ListBoxTags( ListBox ):
             
             menu = QW.QMenu()
             
+            if self._terms_may_have_child_rows:
+                
+                add_it = True
+                
+                if self._child_rows_allowed:
+                    
+                    if len( self._ordered_terms ) == self._total_positional_rows:
+                        
+                        # no parents to hide!
+                        
+                        add_it = False
+                        
+                    
+                    message = 'hide parent rows'
+                    
+                else:
+                    
+                    message = 'show parent rows'
+                    
+                
+                if add_it:
+                    
+                    ClientGUIMenus.AppendMenuItem( menu, message, 'Show/hide parents.', self.SetChildRowsAllowed, not self._child_rows_allowed )
+                    
+                    ClientGUIMenus.AppendSeparator( menu )
+                    
+                
+            
             copy_menu = QW.QMenu( menu )
             
             selected_copyable_tag_strings = self._GetCopyableTagStrings( COPY_SELECTED_TAGS )
@@ -2385,8 +2438,6 @@ class ListBoxTags( ListBox ):
             
             #
             
-            fallback_service_key = self._GetFallbackServiceKey()
-            
             can_launch_sibling_and_parent_dialogs = len( selected_actual_tags ) > 0 and self.can_spawn_new_windows
             can_show_siblings_and_parents = len( selected_actual_tags ) == 1
             
@@ -2505,7 +2556,7 @@ class ListBoxTags( ListBox ):
                             
                             for t_list in service_key_groups_to_tags.values():
                                 
-                                ClientTags.SortTags( CC.SORT_BY_LEXICOGRAPHIC_ASC, t_list )
+                                ClientTagSorting.SortTags( CC.SORT_BY_LEXICOGRAPHIC_ASC, t_list )
                                 
                             
                             service_key_groups = sorted( service_key_groups_to_tags.keys(), key = lambda s_k_g: ( -len( s_k_g ), convert_service_keys_to_name_string( s_k_g ) ) )
@@ -2533,8 +2584,8 @@ class ListBoxTags( ListBox ):
                             
                             for ( t_list_1, t_list_2 ) in service_key_groups_to_tags.values():
                                 
-                                ClientTags.SortTags( CC.SORT_BY_LEXICOGRAPHIC_ASC, t_list_1 )
-                                ClientTags.SortTags( CC.SORT_BY_LEXICOGRAPHIC_ASC, t_list_2 )
+                                ClientTagSorting.SortTags( CC.SORT_BY_LEXICOGRAPHIC_ASC, t_list_1 )
+                                ClientTagSorting.SortTags( CC.SORT_BY_LEXICOGRAPHIC_ASC, t_list_2 )
                                 
                             
                             service_key_groups = sorted( service_key_groups_to_tags.keys(), key = lambda s_k_g: ( -len( s_k_g ), convert_service_keys_to_name_string( s_k_g ) ) )
@@ -2636,6 +2687,91 @@ class ListBoxTags( ListBox ):
                 
                 ClientGUIMenus.AppendSeparator( menu )
                 
+                ( predicates, or_predicate, inverse_predicates ) = self._GetSelectedPredicatesAndInverseCopies()
+                
+                if len( predicates ) > 0:
+                    
+                    if self.can_spawn_new_windows or self._CanProvideCurrentPagePredicates():
+                        
+                        search_menu = QW.QMenu( menu )
+                        
+                        ClientGUIMenus.AppendMenu( menu, search_menu, 'search' )
+                        
+                    
+                    if self.can_spawn_new_windows:
+                        
+                        ClientGUIMenus.AppendMenuItem( search_menu, 'open a new search page for ' + selection_string, 'Open a new search page starting with the selected predicates.', self._NewSearchPages, [ predicates ] )
+                        
+                        if or_predicate is not None:
+                            
+                            ClientGUIMenus.AppendMenuItem( search_menu, 'open a new OR search page for ' + selection_string, 'Open a new search page starting with the selected merged as an OR search predicate.', self._NewSearchPages, [ ( or_predicate, ) ] )
+                            
+                        
+                        if len( predicates ) > 1:
+                            
+                            for_each_predicates = [ ( predicate, ) for predicate in predicates ]
+                            
+                            ClientGUIMenus.AppendMenuItem( search_menu, 'open new search pages for each in selection', 'Open one new search page for each selected predicate.', self._NewSearchPages, for_each_predicates )
+                            
+                        
+                        ClientGUIMenus.AppendSeparator( search_menu )
+                        
+                    
+                    if self._CanProvideCurrentPagePredicates():
+                        
+                        current_predicates = self._GetCurrentPagePredicates()
+                        
+                        predicates = set( predicates )
+                        inverse_predicates = set( inverse_predicates )
+                        
+                        if len( predicates ) == 1:
+                            
+                            ( pred, ) = predicates
+                            
+                            predicates_selection_string = pred.ToString( with_count = False )
+                            
+                        else:
+                            
+                            predicates_selection_string = 'selected'
+                            
+                        
+                        some_selected_in_current = HydrusData.SetsIntersect( predicates, current_predicates )
+                        
+                        if some_selected_in_current:
+                            
+                            ClientGUIMenus.AppendMenuItem( search_menu, 'remove {} from current search'.format( predicates_selection_string ), 'Remove the selected predicates from the current search.', self._ProcessMenuPredicateEvent, 'remove_predicates' )
+                            
+                        
+                        some_selected_not_in_current = len( predicates.intersection( current_predicates ) ) < len( predicates )
+                        
+                        if some_selected_not_in_current:
+                            
+                            ClientGUIMenus.AppendMenuItem( search_menu, 'add {} to current search'.format( predicates_selection_string ), 'Add the selected predicates to the current search.', self._ProcessMenuPredicateEvent, 'add_predicates' )
+                            
+                        
+                        if or_predicate is not None:
+                            
+                            ClientGUIMenus.AppendMenuItem( search_menu, 'add an OR of {} to current search'.format( predicates_selection_string ), 'Add the selected predicates as an OR predicate to the current search.', self._ProcessMenuPredicateEvent, 'add_or_predicate' )
+                            
+                        
+                        some_selected_are_excluded_explicitly = HydrusData.SetsIntersect( inverse_predicates, current_predicates )
+                        
+                        if some_selected_are_excluded_explicitly:
+                            
+                            ClientGUIMenus.AppendMenuItem( search_menu, 'permit {} for current search'.format( predicates_selection_string ), 'Stop disallowing the selected predicates from the current search.', self._ProcessMenuPredicateEvent, 'remove_inverse_predicates' )
+                            
+                        
+                        some_selected_are_not_excluded_explicitly = len( inverse_predicates.intersection( current_predicates ) ) < len( inverse_predicates )
+                        
+                        if some_selected_are_not_excluded_explicitly:
+                            
+                            ClientGUIMenus.AppendMenuItem( search_menu, 'exclude {} from current search'.format( predicates_selection_string ), 'Disallow the selected predicates for the current search.', self._ProcessMenuPredicateEvent, 'add_inverse_predicates' )
+                            
+                        
+                    
+                    self._AddEditMenu( menu )
+                    
+                
                 if len( selected_actual_tags ) > 0 and self._page_key is not None:
                     
                     select_menu = QW.QMenu( menu )
@@ -2679,88 +2815,6 @@ class ListBoxTags( ListBox ):
                     ClientGUIMenus.AppendMenu( menu, select_menu, 'select' )
                     
                 
-                ( predicates, or_predicate, inverse_predicates ) = self._GetSelectedPredicatesAndInverseCopies()
-                
-                if len( predicates ) > 0:
-                    
-                    if self.can_spawn_new_windows:
-                        
-                        open_menu = QW.QMenu( menu )
-                        
-                        ClientGUIMenus.AppendMenuItem( open_menu, 'a new search page for ' + selection_string, 'Open a new search page starting with the selected predicates.', self._NewSearchPages, [ predicates ] )
-                        
-                        if or_predicate is not None:
-                            
-                            ClientGUIMenus.AppendMenuItem( open_menu, 'a new OR search page for ' + selection_string, 'Open a new search page starting with the selected merged as an OR search predicate.', self._NewSearchPages, [ ( or_predicate, ) ] )
-                            
-                        
-                        if len( predicates ) > 1:
-                            
-                            for_each_predicates = [ ( predicate, ) for predicate in predicates ]
-                            
-                            ClientGUIMenus.AppendMenuItem( open_menu, 'new search pages for each in selection', 'Open one new search page for each selected predicate.', self._NewSearchPages, for_each_predicates )
-                            
-                        
-                        ClientGUIMenus.AppendMenu( menu, open_menu, 'open' )
-                        
-                    
-                    self._AddEditMenu( menu )
-                    
-                    if self._CanProvideCurrentPagePredicates():
-                        
-                        current_predicates = self._GetCurrentPagePredicates()
-                        
-                        ClientGUIMenus.AppendSeparator( menu )
-                        
-                        predicates = set( predicates )
-                        inverse_predicates = set( inverse_predicates )
-                        
-                        if len( predicates ) == 1:
-                            
-                            ( pred, ) = predicates
-                            
-                            predicates_selection_string = pred.ToString( with_count = False )
-                            
-                        else:
-                            
-                            predicates_selection_string = 'selected'
-                            
-                        
-                        some_selected_in_current = HydrusData.SetsIntersect( predicates, current_predicates )
-                        
-                        if some_selected_in_current:
-                            
-                            ClientGUIMenus.AppendMenuItem( menu, 'remove {} from current search'.format( predicates_selection_string ), 'Remove the selected predicates from the current search.', self._ProcessMenuPredicateEvent, 'remove_predicates' )
-                            
-                        
-                        some_selected_not_in_current = len( predicates.intersection( current_predicates ) ) < len( predicates )
-                        
-                        if some_selected_not_in_current:
-                            
-                            ClientGUIMenus.AppendMenuItem( menu, 'add {} to current search'.format( predicates_selection_string ), 'Add the selected predicates to the current search.', self._ProcessMenuPredicateEvent, 'add_predicates' )
-                            
-                        
-                        if or_predicate is not None:
-                            
-                            ClientGUIMenus.AppendMenuItem( menu, 'add an OR of {} to current search'.format( predicates_selection_string ), 'Add the selected predicates as an OR predicate to the current search.', self._ProcessMenuPredicateEvent, 'add_or_predicate' )
-                            
-                        
-                        some_selected_are_excluded_explicitly = HydrusData.SetsIntersect( inverse_predicates, current_predicates )
-                        
-                        if some_selected_are_excluded_explicitly:
-                            
-                            ClientGUIMenus.AppendMenuItem( menu, 'permit {} for current search'.format( predicates_selection_string ), 'Stop disallowing the selected predicates from the current search.', self._ProcessMenuPredicateEvent, 'remove_inverse_predicates' )
-                            
-                        
-                        some_selected_are_not_excluded_explicitly = len( inverse_predicates.intersection( current_predicates ) ) < len( inverse_predicates )
-                        
-                        if some_selected_are_not_excluded_explicitly:
-                            
-                            ClientGUIMenus.AppendMenuItem( menu, 'exclude {} from current search'.format( predicates_selection_string ), 'Disallow the selected predicates for the current search.', self._ProcessMenuPredicateEvent, 'add_inverse_predicates' )
-                            
-                        
-                    
-                
             
             if len( selected_actual_tags ) == 1:
                 
@@ -2779,10 +2833,6 @@ class ListBoxTags( ListBox ):
                     
                     ClientGUIMenus.AppendMenu( menu, hide_menu, 'hide' )
                     
-                    ClientGUIMenus.AppendSeparator( menu )
-                    
-                
-                ClientGUIMenus.AppendSeparator( menu )
                 
                 def set_favourite_tags( tag ):
                     
@@ -2815,7 +2865,11 @@ class ListBoxTags( ListBox ):
                     description = 'Add this tag from your favourites'
                     
                 
-                ClientGUIMenus.AppendMenuItem( menu, label, description, set_favourite_tags, selected_tag )
+                favourites_menu = QW.QMenu( menu )
+                
+                ClientGUIMenus.AppendMenuItem( favourites_menu, label, description, set_favourite_tags, selected_tag )
+                
+                m = ClientGUIMenus.AppendMenu( menu, favourites_menu, 'favourites' )
                 
             
             CGC.core().PopupMenu( self, menu )
@@ -2829,16 +2883,14 @@ class ListBoxTags( ListBox ):
     
 class ListBoxTagsPredicates( ListBoxTags ):
     
-    def __init__( self, *args, render_for_user = True, **kwargs ):
+    def __init__( self, *args, tag_display_type = ClientTags.TAG_DISPLAY_ACTUAL, **kwargs ):
         
-        ListBoxTags.__init__( self, *args, render_for_user = render_for_user, **kwargs )
+        ListBoxTags.__init__( self, *args, tag_display_type = tag_display_type, **kwargs )
         
     
     def _GenerateTermFromPredicate( self, predicate: ClientSearch.Predicate ) -> ClientGUIListBoxesData.ListBoxItemPredicate:
         
-        show_ideal_siblings = self._tag_display_type == ClientTags.TAG_DISPLAY_STORAGE
-        
-        return ClientGUIListBoxesData.ListBoxItemPredicate( predicate, show_ideal_siblings )
+        return ClientGUIListBoxesData.ListBoxItemPredicate( predicate )
         
     
     def _GetMutuallyExclusivePredicates( self, predicate ):
@@ -3082,7 +3134,7 @@ class ListBoxTagsFilter( ListBoxTags ):
     
 class ListBoxTagsDisplayCapable( ListBoxTags ):
     
-    def __init__( self, parent, service_key = None, show_display_decorators = True, render_for_user = True, **kwargs ):
+    def __init__( self, parent, service_key = None, tag_display_type = ClientTags.TAG_DISPLAY_ACTUAL, **kwargs ):
         
         if service_key is None:
             
@@ -3090,43 +3142,46 @@ class ListBoxTagsDisplayCapable( ListBoxTags ):
             
         
         self._service_key = service_key
-        self._show_display_decorators = show_display_decorators
         
-        has_async_text_info = self._show_display_decorators
+        has_async_text_info = tag_display_type == ClientTags.TAG_DISPLAY_STORAGE
         
-        ListBoxTags.__init__( self, parent, has_async_text_info = has_async_text_info, render_for_user = render_for_user, **kwargs )
+        ListBoxTags.__init__( self, parent, has_async_text_info = has_async_text_info, tag_display_type = tag_display_type, **kwargs )
         
     
-    def _ApplyAsyncInfoToTerm( self, term, info ):
+    def _ApplyAsyncInfoToTerm( self, term, info ) -> typing.Tuple[ bool, bool ]:
         
         # this guy comes with the lock
         
         if info is None:
             
-            return
+            return ( False, False )
             
         
+        sort_info_changed = False
+        num_rows_changed = False
+        
         ( ideal, parents ) = info
         
         if ideal is not None and ideal != term.GetTag():
             
             term.SetIdealTag( ideal )
             
+            sort_info_changed = True
+            
         
         if parents is not None:
             
             term.SetParents( parents )
             
+            num_rows_changed = True
+            
         
-    
-    def _GetFallbackServiceKey( self ):
-        
-        return self._service_key
+        return ( sort_info_changed, num_rows_changed )
         
     
     def _InitialiseAsyncTextInfoUpdaterWorkCallable( self ):
         
-        if not self._show_display_decorators:
+        if not self._has_async_text_info:
             
             return ListBoxTags._InitialiseAsyncTextInfoUpdaterWorkCallable( self )
             
@@ -3191,11 +3246,11 @@ class ListBoxTagsDisplayCapable( ListBoxTags ):
     
 class ListBoxTagsStrings( ListBoxTagsDisplayCapable ):
     
-    def __init__( self, parent, service_key = None, show_display_decorators = True, sort_tags = True, **kwargs ):
+    def __init__( self, parent, service_key = None, sort_tags = True, **kwargs ):
         
         self._sort_tags = sort_tags
         
-        ListBoxTagsDisplayCapable.__init__( self, parent, service_key = service_key, show_display_decorators = show_display_decorators, **kwargs )
+        ListBoxTagsDisplayCapable.__init__( self, parent, service_key = service_key, **kwargs )
         
     
     def _GenerateTermFromTag( self, tag: str ) -> ClientGUIListBoxesData.ListBoxItemTextTag:
@@ -3238,9 +3293,9 @@ class ListBoxTagsStrings( ListBoxTagsDisplayCapable ):
     
 class ListBoxTagsStringsAddRemove( ListBoxTagsStrings ):
     
-    def __init__( self, parent, service_key = None, removed_callable = None, show_display_decorators = True ):
+    def __init__( self, parent, service_key, tag_display_type, removed_callable = None ):
         
-        ListBoxTagsStrings.__init__( self, parent, service_key = service_key, show_display_decorators = show_display_decorators )
+        ListBoxTagsStrings.__init__( self, parent, service_key = service_key, tag_display_type = tag_display_type )
         
         self._removed_callable = removed_callable
         
@@ -3348,21 +3403,19 @@ class ListBoxTagsStringsAddRemove( ListBoxTagsStrings ):
     
 class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
     
-    def __init__( self, parent, tag_display_type, service_key = None, show_display_decorators = False, include_counts = True, render_for_user = True ):
+    def __init__( self, parent, tag_display_type, service_key = None, include_counts = True ):
         
         if service_key is None:
             
             service_key = CC.COMBINED_TAG_SERVICE_KEY
             
         
-        ListBoxTagsDisplayCapable.__init__( self, parent, service_key = service_key, show_display_decorators = show_display_decorators, render_for_user = render_for_user, height_num_chars = 24 )
+        ListBoxTagsDisplayCapable.__init__( self, parent, service_key = service_key, tag_display_type = tag_display_type, height_num_chars = 24 )
         
         self._sort = HC.options[ 'default_tag_sort' ]
         
         self._last_media = set()
         
-        self._tag_display_type = tag_display_type
-        
         self._include_counts = include_counts
         
         self._current_tags_to_count = collections.Counter()
@@ -3449,16 +3502,38 @@ class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
     
     def _Sort( self ):
         
-        tags_to_count = collections.Counter()
+        # I do this weird terms to count instead of tags to count because of tag vs ideal tag gubbins later on in sort
         
-        if self._show_current: tags_to_count.update( self._current_tags_to_count )
-        if self._show_deleted: tags_to_count.update( self._deleted_tags_to_count )
-        if self._show_pending: tags_to_count.update( self._pending_tags_to_count )
-        if self._show_petitioned: tags_to_count.update( self._petitioned_tags_to_count )
+        terms_to_count = collections.Counter()
         
-        item_to_tag_key_wrapper = lambda term: term.GetTag()
+        jobs = [
+            ( self._show_current, self._current_tags_to_count ),
+            ( self._show_deleted, self._deleted_tags_to_count ),
+            ( self._show_pending, self._pending_tags_to_count ),
+            ( self._show_petitioned, self._petitioned_tags_to_count )
+        ]
         
-        ClientTags.SortTags( self._sort, self._ordered_terms, tags_to_count = tags_to_count, item_to_tag_key_wrapper = item_to_tag_key_wrapper )
+        counts_to_include = [ c for ( show, c ) in jobs ]
+        
+        for term in self._ordered_terms:
+            
+            tag = term.GetTag()
+            
+            count = sum( ( c[ tag ] for c in counts_to_include if tag in c ) )
+            
+            terms_to_count[ term ] = count
+            
+        
+        if self._sibling_decoration_allowed:
+            
+            item_to_tag_key_wrapper = lambda term: term.GetBestTag()
+            
+        else:
+            
+            item_to_tag_key_wrapper = lambda term: term.GetTag()
+            
+        
+        ClientTagSorting.SortTags( self._sort, self._ordered_terms, tag_items_to_count = terms_to_count, item_to_tag_key_wrapper = item_to_tag_key_wrapper )
         
         self._RegenTermsToIndices()
         
@@ -3676,7 +3751,7 @@ class ListBoxTagsMediaTagsDialog( ListBoxTagsMedia ):
     
     def __init__( self, parent, enter_func, delete_func ):
         
-        ListBoxTagsMedia.__init__( self, parent, ClientTags.TAG_DISPLAY_STORAGE, render_for_user = False, show_display_decorators = True, include_counts = True )
+        ListBoxTagsMedia.__init__( self, parent, ClientTags.TAG_DISPLAY_STORAGE, include_counts = True )
         
         self._enter_func = enter_func
         self._delete_func = delete_func

hydrus/client/gui/lists/ClientGUIListBoxesData.py

@@ -56,12 +56,12 @@ class ListBoxItem( object ):
         raise NotImplementedError()
         
     
-    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
+    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool, sibling_decoration_allowed: bool, child_rows_allowed: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
         
         raise NotImplementedError()
         
     
-    def GetRowCount( self ):
+    def GetRowCount( self, child_rows_allowed: bool ):
         
         return 1
         
@@ -95,7 +95,7 @@ class ListBoxItemTagSlice( ListBoxItem ):
         return []
         
     
-    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool ) -> typing.List[ typing.Tuple[ str, str ] ]:
+    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool, sibling_decoration_allowed: bool, child_rows_allowed: bool ) -> typing.List[ typing.Tuple[ str, str ] ]:
         
         presentation_text = self.GetCopyableText()
         
@@ -167,7 +167,7 @@ class ListBoxItemNamespaceColour( ListBoxItem ):
         return []
         
     
-    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
+    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool, sibling_decoration_allowed: bool, child_rows_allowed: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
         
         return [ [ ( self.GetCopyableText(), self._namespace ) ] ]
         
@@ -218,6 +218,16 @@ class ListBoxItemTextTag( ListBoxItem ):
             
         
     
+    def GetBestTag( self ) -> str:
+        
+        if self._ideal_tag is None:
+            
+            return self._tag
+            
+        
+        return self._ideal_tag
+        
+    
     def GetCopyableText( self, with_counts: bool = False ) -> str:
         
         return self._tag
@@ -228,9 +238,9 @@ class ListBoxItemTextTag( ListBoxItem ):
         return [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, self._tag ) ]
         
     
-    def GetRowCount( self ):
+    def GetRowCount( self, child_rows_allowed: bool ):
         
-        if self._parent_tags is None:
+        if self._parent_tags is None or not child_rows_allowed:
             
             return 1
             
@@ -240,7 +250,7 @@ class ListBoxItemTextTag( ListBoxItem ):
             
         
     
-    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
+    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool, sibling_decoration_allowed: bool, child_rows_allowed: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
         
         # this should be with counts or whatever, but we need to think about this more lad
         
@@ -250,14 +260,14 @@ class ListBoxItemTextTag( ListBoxItem ):
         
         texts_with_namespaces = [ ( tag_text, namespace ) ]
         
-        if self._ideal_tag is not None:
+        if sibling_decoration_allowed and self._ideal_tag is not None:
             
             self._AppendIdealTagTextWithNamespace( texts_with_namespaces, render_for_user )
             
         
         rows_of_texts_with_namespaces = [ texts_with_namespaces ]
         
-        if self._parent_tags is not None:
+        if child_rows_allowed and self._parent_tags is not None:
             
             self._AppendParentsTextWithNamespaces( rows_of_texts_with_namespaces, render_for_user )
             
@@ -314,7 +324,7 @@ class ListBoxItemTextTagWithCounts( ListBoxItemTextTag ):
         
         if with_counts:
             
-            return ''.join( ( text for ( text, namespace ) in self.GetRowsOfPresentationTextsWithNamespaces( False )[0] ) )
+            return ''.join( ( text for ( text, namespace ) in self.GetRowsOfPresentationTextsWithNamespaces( False, False, False )[0] ) )
             
         else:
             
@@ -329,7 +339,7 @@ class ListBoxItemTextTagWithCounts( ListBoxItemTextTag ):
         return [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, self._tag ) ]
         
     
-    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
+    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool, sibling_decoration_allowed: bool, child_rows_allowed: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
         
         # this should be with counts or whatever, but we need to think about this more lad
         
@@ -379,14 +389,14 @@ class ListBoxItemTextTagWithCounts( ListBoxItemTextTag ):
         
         texts_with_namespaces = [ ( tag_text, namespace ) ]
         
-        if self._ideal_tag is not None:
+        if sibling_decoration_allowed and self._ideal_tag is not None:
             
             self._AppendIdealTagTextWithNamespace( texts_with_namespaces, render_for_user )
             
         
         rows_of_texts_with_namespaces = [ texts_with_namespaces ]
         
-        if self._parent_tags is not None:
+        if child_rows_allowed and self._parent_tags is not None:
             
             self._AppendParentsTextWithNamespaces( rows_of_texts_with_namespaces, render_for_user )
             
@@ -396,12 +406,11 @@ class ListBoxItemTextTagWithCounts( ListBoxItemTextTag ):
     
 class ListBoxItemPredicate( ListBoxItem ):
     
-    def __init__( self, predicate: ClientSearch.Predicate, show_ideal_siblings: bool ):
+    def __init__( self, predicate: ClientSearch.Predicate ):
         
         ListBoxItem.__init__( self )
         
         self._predicate = predicate
-        self._show_ideal_siblings = show_ideal_siblings
         self._i_am_an_or_under_construction = False
         
     
@@ -443,18 +452,25 @@ class ListBoxItemPredicate( ListBoxItem ):
             
         
     
-    def GetRowCount( self ):
+    def GetRowCount( self, child_rows_allowed: bool ):
         
-        return 1 + len( self._predicate.GetParentPredicates() )
+        if child_rows_allowed:
+            
+            return 1 + len( self._predicate.GetParentPredicates() )
+            
+        else:
+            
+            return 1
+            
         
     
-    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
+    def GetRowsOfPresentationTextsWithNamespaces( self, render_for_user: bool, sibling_decoration_allowed: bool, child_rows_allowed: bool ) -> typing.List[ typing.List[ typing.Tuple[ str, str ] ] ]:
         
         rows_of_texts_and_namespaces = []
         
         texts_and_namespaces = self._predicate.GetTextsAndNamespaces( render_for_user, or_under_construction = self._i_am_an_or_under_construction )
         
-        if self._show_ideal_siblings and self._predicate.HasIdealSibling():
+        if sibling_decoration_allowed and self._predicate.HasIdealSibling():
             
             ideal_sibling = self._predicate.GetIdealSibling()
             
@@ -467,9 +483,12 @@ class ListBoxItemPredicate( ListBoxItem ):
         
         rows_of_texts_and_namespaces.append( texts_and_namespaces )
         
-        for parent_pred in self._predicate.GetParentPredicates():
+        if child_rows_allowed:
             
-            rows_of_texts_and_namespaces.append( parent_pred.GetTextsAndNamespaces( render_for_user ) )
+            for parent_pred in self._predicate.GetParentPredicates():
+                
+                rows_of_texts_and_namespaces.append( parent_pred.GetTextsAndNamespaces( render_for_user ) )
+                
             
         
         return rows_of_texts_and_namespaces

hydrus/client/gui/search/ClientGUIACDropdown.py

@@ -419,7 +419,7 @@ def ShouldDoExactSearch( parsed_autocomplete_text: ClientSearch.ParsedAutocomple
     
     return len( test_text ) <= exact_match_character_threshold
     
-def WriteFetch( win, job_key, results_callable, parsed_autocomplete_text: ClientSearch.ParsedAutocompleteText, tag_search_context: ClientSearch.TagSearchContext, file_service_key: bytes, expand_parents: bool, results_cache: ClientSearch.PredicateResultsCache ):
+def WriteFetch( win, job_key, results_callable, parsed_autocomplete_text: ClientSearch.ParsedAutocompleteText, tag_search_context: ClientSearch.TagSearchContext, file_service_key: bytes, results_cache: ClientSearch.PredicateResultsCache ):
     
     display_tag_service_key = tag_search_context.display_service_key
     
@@ -505,7 +505,7 @@ def WriteFetch( win, job_key, results_callable, parsed_autocomplete_text: Client
     
     HG.client_controller.CallLaterQtSafe( win, 0.0, results_callable, job_key, parsed_autocomplete_text, results_cache, matches )
     
-class ListBoxTagsAC( ClientGUIListBoxes.ListBoxTagsPredicates ):
+class ListBoxTagsPredicatesAC( ClientGUIListBoxes.ListBoxTagsPredicates ):
     
     def __init__( self, parent, callable, service_key, float_mode, **kwargs ):
         
@@ -543,6 +543,18 @@ class ListBoxTagsAC( ClientGUIListBoxes.ListBoxTagsPredicates ):
         return False
         
     
+    def _GenerateTermFromPredicate( self, predicate: ClientSearch.Predicate ):
+        
+        term = ClientGUIListBoxes.ListBoxTagsPredicates._GenerateTermFromPredicate( self, predicate )
+        
+        if predicate.GetType() == ClientSearch.PREDICATE_TYPE_OR_CONTAINER:
+            
+            term.SetORUnderConstruction( True )
+            
+        
+        return term
+        
+    
     def SetPredicates( self, predicates ):
         
         # need to do a clever compare, since normal predicate compare doesn't take count into account
@@ -618,30 +630,44 @@ class ListBoxTagsAC( ClientGUIListBoxes.ListBoxTagsPredicates ):
             
         
     
-    def SetTagService( self, service_key ):
+    def SetTagServiceKey( self, service_key: bytes ):
         
         self._service_key = service_key
         
     
-class ListBoxTagsACRead( ListBoxTagsAC ):
+class ListBoxTagsStringsAC( ClientGUIListBoxes.ListBoxTagsStrings ):
     
-    def _GenerateTermFromPredicate( self, predicate: ClientSearch.Predicate ):
+    def __init__( self, parent, callable, service_key, float_mode, **kwargs ):
         
-        term = ListBoxTagsAC._GenerateTermFromPredicate( self, predicate )
+        ClientGUIListBoxes.ListBoxTagsStrings.__init__( self, parent, service_key = service_key, sort_tags = False, **kwargs )
         
-        if predicate.GetType() == ClientSearch.PREDICATE_TYPE_OR_CONTAINER:
+        self._callable = callable
+        self._float_mode = float_mode
+        
+    
+    def _Activate( self, shift_down ) -> bool:
+        
+        predicates = self._GetPredicatesFromTerms( self._selected_terms )
+        
+        if self._float_mode:
             
-            term.SetORUnderConstruction( True )
+            widget = self.window().parentWidget()
+            
+        else:
+            
+            widget = self
             
         
-        return term
+        predicates = ClientGUISearch.FleshOutPredicates( widget, predicates )
         
-    
-class ListBoxTagsACWrite( ListBoxTagsAC ):
-    
-    def __init__( self, *args, render_for_user = False, **kwargs ):
+        if len( predicates ) > 0:
+            
+            self._callable( predicates, shift_down )
+            
+            return True
+            
         
-        ListBoxTagsAC.__init__( self, *args, render_for_user = render_for_user, **kwargs )
+        return False
         
     
 # much of this is based on the excellent TexCtrlAutoComplete class by Edward Flick, Michele Petrazzo and Will Sadkin, just with plenty of simplification and integration into hydrus
@@ -1396,7 +1422,8 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
         
         self._tag_service_key = tag_service_key
         
-        self._search_results_list.SetTagService( self._tag_service_key )
+        self._search_results_list.SetTagServiceKey( self._tag_service_key )
+        self._favourites_list.SetTagServiceKey( self._tag_service_key )
         
         self._UpdateTagServiceLabel()
         
@@ -1492,8 +1519,8 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
     
     def NotifyNewServices( self ):
         
-        self.SetFileService( self._file_service_key )
-        self.SetTagService( self._tag_service_key )
+        self.SetFileServiceKey( self._file_service_key )
+        self.SetTagServiceKey( self._tag_service_key )
         
     
     def RefreshFavouriteTags( self ):
@@ -1505,7 +1532,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
         self._favourites_list.SetPredicates( predicates )
         
     
-    def SetFileService( self, file_service_key ):
+    def SetFileServiceKey( self, file_service_key ):
         
         self._ChangeFileService( file_service_key )
         
@@ -1518,7 +1545,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
             
         
     
-    def SetTagService( self, tag_service_key ):
+    def SetTagServiceKey( self, tag_service_key ):
         
         self._ChangeTagService( tag_service_key )
         
@@ -1847,7 +1874,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
         
         height_num_chars = HG.client_controller.new_options.GetInteger( 'ac_read_list_height_num_chars' )
         
-        favs_list = ListBoxTagsACRead( self._dropdown_notebook, self.BroadcastChoices, self._float_mode, self._tag_service_key, height_num_chars = height_num_chars )
+        favs_list = ListBoxTagsPredicatesAC( self._dropdown_notebook, self.BroadcastChoices, self._float_mode, self._tag_service_key, tag_display_type = ClientTags.TAG_DISPLAY_ACTUAL, height_num_chars = height_num_chars )
         
         return favs_list
         
@@ -1856,7 +1883,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
         
         height_num_chars = HG.client_controller.new_options.GetInteger( 'ac_read_list_height_num_chars' )
         
-        return ListBoxTagsACRead( self._dropdown_notebook, self.BroadcastChoices, self._tag_service_key, self._float_mode, height_num_chars = height_num_chars )
+        return ListBoxTagsPredicatesAC( self._dropdown_notebook, self.BroadcastChoices, self._tag_service_key, self._float_mode, tag_display_type = ClientTags.TAG_DISPLAY_ACTUAL, height_num_chars = height_num_chars )
         
     
     def _LoadFavouriteSearch( self, folder_name, name ):
@@ -2332,7 +2359,9 @@ class ListBoxTagsActiveSearchPredicates( ClientGUIListBoxes.ListBoxTagsPredicate
                     
                     terms_to_be_added.add( term )
                     
-                    terms_to_be_removed.update( self._GetMutuallyExclusivePredicates( predicate ) )
+                    m_e_preds = self._GetMutuallyExclusivePredicates( predicate )
+                    
+                    terms_to_be_removed.update( ( self._GenerateTermFromPredicate( pred ) for pred in m_e_preds ) )
                     
                 
             
@@ -2413,12 +2442,11 @@ class ListBoxTagsActiveSearchPredicates( ClientGUIListBoxes.ListBoxTagsPredicate
     
 class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
     
-    def __init__( self, parent, chosen_tag_callable, expand_parents, file_service_key, tag_service_key, null_entry_callable = None, tag_service_key_changed_callable = None, show_paste_button = False ):
+    def __init__( self, parent, chosen_tag_callable, file_service_key, tag_service_key, null_entry_callable = None, tag_service_key_changed_callable = None, show_paste_button = False ):
         
         self._display_tag_service_key = tag_service_key
         
         self._chosen_tag_callable = chosen_tag_callable
-        self._expand_parents = expand_parents
         self._null_entry_callable = null_entry_callable
         self._tag_service_key_changed_callable = tag_service_key_changed_callable
         
@@ -2502,7 +2530,9 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
         
         height_num_chars = HG.client_controller.new_options.GetInteger( 'ac_write_list_height_num_chars' )
         
-        favs_list = ListBoxTagsACWrite( self._dropdown_notebook, self.BroadcastChoices, self._display_tag_service_key, self._float_mode, height_num_chars = height_num_chars )
+        favs_list = ListBoxTagsStringsAC( self._dropdown_notebook, self.BroadcastChoices, self._display_tag_service_key, self._float_mode, tag_display_type = ClientTags.TAG_DISPLAY_STORAGE, height_num_chars = height_num_chars )
+        
+        favs_list.SetChildRowsAllowed( HG.client_controller.new_options.GetBoolean( 'expand_parents_on_storage_autocomplete_taglists' ) )
         
         return favs_list
         
@@ -2511,7 +2541,11 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
         
         height_num_chars = HG.client_controller.new_options.GetInteger( 'ac_write_list_height_num_chars' )
         
-        return ListBoxTagsACWrite( self._dropdown_notebook, self.BroadcastChoices, self._display_tag_service_key, self._float_mode, height_num_chars = height_num_chars )
+        preds_list = ListBoxTagsPredicatesAC( self._dropdown_notebook, self.BroadcastChoices, self._display_tag_service_key, self._float_mode, tag_display_type = ClientTags.TAG_DISPLAY_STORAGE, height_num_chars = height_num_chars )
+        
+        preds_list.SetChildRowsAllowed( HG.client_controller.new_options.GetBoolean( 'expand_parents_on_storage_autocomplete_taglists' ) )
+        
+        return preds_list
         
     
     def _Paste( self ):
@@ -2586,7 +2620,7 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
         
         tag_search_context = ClientSearch.TagSearchContext( service_key = self._tag_service_key, display_service_key = self._display_tag_service_key )
         
-        HG.client_controller.CallToThread( WriteFetch, self, job_key, self.SetFetchedResults, parsed_autocomplete_text, tag_search_context, self._file_service_key, self._expand_parents, self._results_cache )
+        HG.client_controller.CallToThread( WriteFetch, self, job_key, self.SetFetchedResults, parsed_autocomplete_text, tag_search_context, self._file_service_key, self._results_cache )
         
     
     def _TakeResponsibilityForEnter( self, shift_down ):
@@ -2615,9 +2649,7 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
         
         favourite_tags = sorted( HG.client_controller.new_options.GetStringList( 'favourite_tags' ) )
         
-        predicates = [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, tag ) for tag in favourite_tags ]
-        
-        self._favourites_list.SetPredicates( predicates )
+        self._favourites_list.SetTags( favourite_tags )
         
     
     def SetDisplayTagServiceKey( self, service_key ):
@@ -2625,11 +2657,6 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
         self._display_tag_service_key = service_key
         
     
-    def SetExpandParents( self, expand_parents ):
-        
-        self._expand_parents = expand_parents
-        
-    
 class EditAdvancedORPredicates( ClientGUIScrolledPanels.EditPanel ):
     
     def __init__( self, parent, initial_string = None ):

hydrus/client/importing/ClientImportFileSeeds.py

@@ -1143,10 +1143,14 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
                     
                     status_hook( 'downloading file page' )
                     
-                    if self._referral_url not in ( post_url, url_to_check ):
+                    if self._referral_url is not None and self._referral_url != url_to_check:
                         
                         referral_url = self._referral_url
                         
+                    elif url_to_check != post_url:
+                        
+                        referral_url = post_url
+                        
                     else:
                         
                         referral_url = None

hydrus/client/importing/ClientImportGallerySeeds.py

@@ -338,10 +338,14 @@ class GallerySeed( HydrusSerialisable.SerialisableBase ):
             
             status_hook( 'downloading gallery page' )
             
-            if self._referral_url not in ( gallery_url, url_to_check ):
+            if self._referral_url is not None and self._referral_url != url_to_check:
                 
                 referral_url = self._referral_url
                 
+            elif gallery_url != url_to_check:
+                
+                referral_url = gallery_url
+                
             else:
                 
                 referral_url = None

hydrus/client/metadata/ClientTagSorting.py

@@ -0,0 +1,150 @@
+from hydrus.core import HydrusTags
+
+from hydrus.client import ClientConstants as CC
+
+def SortTags( sort_by, list_of_tag_items, tag_items_to_count = None, item_to_tag_key_wrapper = None ):
+    
+    def lexicographic_key( tag ):
+        
+        ( namespace, subtag ) = HydrusTags.SplitTag( tag )
+        
+        comparable_namespace = HydrusTags.ConvertTagToSortable( namespace )
+        comparable_subtag = HydrusTags.ConvertTagToSortable( subtag )
+        
+        if namespace == '':
+            
+            return ( comparable_subtag, comparable_subtag )
+            
+        else:
+            
+            return ( comparable_namespace, comparable_subtag )
+            
+        
+    
+    def subtag_lexicographic_key( tag ):
+        
+        ( namespace, subtag ) = HydrusTags.SplitTag( tag )
+        
+        comparable_subtag = HydrusTags.ConvertTagToSortable( subtag )
+        
+        return comparable_subtag
+        
+    
+    def incidence_key( tag_item ):
+        
+        if tag_items_to_count is None:
+            
+            return 1
+            
+        else:
+            
+            return tag_items_to_count[ tag_item ]
+            
+        
+    
+    def namespace_key( tag ):
+        
+        ( namespace, subtag ) = HydrusTags.SplitTag( tag )
+        
+        if namespace == '':
+            
+            namespace = '{' # '{' is above 'z' in ascii, so this works for most situations
+            
+        
+        return namespace
+        
+    
+    def namespace_lexicographic_key( tag ):
+        
+        # '{' is above 'z' in ascii, so this works for most situations
+        
+        ( namespace, subtag ) = HydrusTags.SplitTag( tag )
+        
+        if namespace == '':
+            
+            return ( '{', HydrusTags.ConvertTagToSortable( subtag ) )
+            
+        else:
+            
+            return ( namespace, HydrusTags.ConvertTagToSortable( subtag ) )
+            
+        
+    
+    sorts_to_do = []
+    
+    if sort_by in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
+        
+        # let's establish a-z here for equal incidence values later
+        if sort_by in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC ):
+            
+            sorts_to_do.append( ( lexicographic_key, True ) )
+            
+            reverse = False
+            
+        elif sort_by in ( CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
+            
+            sorts_to_do.append( ( lexicographic_key, False ) )
+            
+            reverse = True
+            
+        
+        sorts_to_do.append( ( incidence_key, reverse ) )
+        
+        if sort_by in ( CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
+            
+            # python list sort is stable, so lets now sort again
+            
+            if sort_by == CC.SORT_BY_INCIDENCE_NAMESPACE_ASC:
+                
+                reverse = True
+                
+            elif sort_by == CC.SORT_BY_INCIDENCE_NAMESPACE_DESC:
+                
+                reverse = False
+                
+            
+            sorts_to_do.append( ( namespace_key, reverse ) )
+            
+        
+    else:
+        
+        if sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_DESC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC, CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC ):
+            
+            reverse = True
+            
+        elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC ):
+            
+            reverse = False
+            
+        
+        if sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
+            
+            key = namespace_lexicographic_key
+            
+        elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_DESC ):
+            
+            key = lexicographic_key
+            
+        elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC ):
+            
+            key = subtag_lexicographic_key
+            
+        
+        sorts_to_do.append( ( key, reverse ) )
+        
+    
+    for ( key, reverse ) in sorts_to_do:
+        
+        key_to_use = key
+        
+        if key_to_use != incidence_key: # other keys use tag, incidence uses tag item
+            
+            if item_to_tag_key_wrapper is not None:
+                
+                key_to_use = lambda item: key( item_to_tag_key_wrapper( item ) )
+                
+            
+        
+        list_of_tag_items.sort( key = key_to_use, reverse = reverse )
+        
+    

hydrus/client/metadata/ClientTags.py

@@ -78,149 +78,6 @@ def RenderTag( tag, render_for_user: bool ):
         return namespace + connector + subtag
         
     
-def SortTags( sort_by, tags_list, tags_to_count = None, item_to_tag_key_wrapper = None ):
-    
-    def lexicographic_key( tag ):
-        
-        ( namespace, subtag ) = HydrusTags.SplitTag( tag )
-        
-        comparable_namespace = HydrusTags.ConvertTagToSortable( namespace )
-        comparable_subtag = HydrusTags.ConvertTagToSortable( subtag )
-        
-        if namespace == '':
-            
-            return ( comparable_subtag, comparable_subtag )
-            
-        else:
-            
-            return ( comparable_namespace, comparable_subtag )
-            
-        
-    
-    def subtag_lexicographic_key( tag ):
-        
-        ( namespace, subtag ) = HydrusTags.SplitTag( tag )
-        
-        comparable_subtag = HydrusTags.ConvertTagToSortable( subtag )
-        
-        return comparable_subtag
-        
-    
-    def incidence_key( tag ):
-        
-        if tags_to_count is None:
-            
-            return 1
-            
-        else:
-            
-            return tags_to_count[ tag ]
-            
-        
-    
-    def namespace_key( tag ):
-        
-        ( namespace, subtag ) = HydrusTags.SplitTag( tag )
-        
-        if namespace == '':
-            
-            namespace = '{' # '{' is above 'z' in ascii, so this works for most situations
-            
-        
-        return namespace
-        
-    
-    def namespace_lexicographic_key( tag ):
-        
-        # '{' is above 'z' in ascii, so this works for most situations
-        
-        ( namespace, subtag ) = HydrusTags.SplitTag( tag )
-        
-        if namespace == '':
-            
-            return ( '{', HydrusTags.ConvertTagToSortable( subtag ) )
-            
-        else:
-            
-            return ( namespace, HydrusTags.ConvertTagToSortable( subtag ) )
-            
-        
-    
-    sorts_to_do = []
-    
-    if sort_by in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
-        
-        # let's establish a-z here for equal incidence values later
-        if sort_by in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC ):
-            
-            sorts_to_do.append( ( lexicographic_key, True ) )
-            
-            reverse = False
-            
-        elif sort_by in ( CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
-            
-            sorts_to_do.append( ( lexicographic_key, False ) )
-            
-            reverse = True
-            
-        
-        sorts_to_do.append( ( incidence_key, reverse ) )
-        
-        if sort_by in ( CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
-            
-            # python list sort is stable, so lets now sort again
-            
-            if sort_by == CC.SORT_BY_INCIDENCE_NAMESPACE_ASC:
-                
-                reverse = True
-                
-            elif sort_by == CC.SORT_BY_INCIDENCE_NAMESPACE_DESC:
-                
-                reverse = False
-                
-            
-            sorts_to_do.append( ( namespace_key, reverse ) )
-            
-        
-    else:
-        
-        if sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_DESC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC, CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC ):
-            
-            reverse = True
-            
-        elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC ):
-            
-            reverse = False
-            
-        
-        if sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
-            
-            key = namespace_lexicographic_key
-            
-        elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_DESC ):
-            
-            key = lexicographic_key
-            
-        elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC ):
-            
-            key = subtag_lexicographic_key
-            
-        
-        sorts_to_do.append( ( key, reverse ) )
-        
-    
-    for ( key, reverse ) in sorts_to_do:
-        
-        key_to_use = key
-        
-        if item_to_tag_key_wrapper is not None:
-            
-            key_to_use = lambda item: key( item_to_tag_key_wrapper( item ) )
-            
-        
-        tags_list.sort( key = key_to_use, reverse = reverse )
-        
-    
 class ServiceKeysToTags( HydrusSerialisable.SerialisableBase, collections.defaultdict ):
     
     SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_SERVICE_KEYS_TO_TAGS

hydrus/client/networking/ClientNetworkingDomain.py

@@ -2275,21 +2275,10 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
     @staticmethod
     def STATICSortURLClassesDescendingComplexity( url_classes ):
         
-        # we sort them in descending complexity so that
-        # site.com/post/123456
-        # comes before
-        # site.com/search?query=blah
+        # sort reverse = true so most complex come first
         
-        # I used to do gallery first, then post, then file, but it ultimately was unhelpful in some situations and better handled by strict component/parameter matching
-        
-        def key( u_m ):
-            
-            u_e = u_m.GetExampleURL()
-            
-            return ( u_e.count( '/' ), u_e.count( '=' ), len( u_e ) )
-            
-        
-        url_classes.sort( key = key, reverse = True )
+        # ( num_path_components, num_required_parameters, num_total_parameters, len_example_url )
+        url_classes.sort( key = lambda u_c: u_c.GetSortingComplexityKey(), reverse = True )
         
     
 HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER ] = NetworkDomainManager
@@ -3453,6 +3442,24 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
         return 'URL Class "' + self._name + '" - ' + ConvertURLIntoDomain( self.GetExampleURL() )
         
     
+    def GetSortingComplexityKey( self ):
+        
+        # we sort url classes so that
+        # site.com/post/123456
+        # comes before
+        # site.com/search?query=blah
+        
+        # I used to do gallery first, then post, then file, but it ultimately was unhelpful in some situations and better handled by strict component/parameter matching
+        
+        num_required_path_components = len( [ 1 for ( string_match, default ) in self._path_components if default is None ] )
+        num_total_path_components = len( self._path_components )
+        num_required_parameters = len( [ 1 for ( key, ( string_match, default ) ) in self._parameters.items() if default is None ] )
+        num_total_parameters = len( self._parameters )
+        len_example_url = len( self.Normalise( self._example_url ) )
+        
+        return ( num_required_parameters, num_total_path_components, num_required_parameters, num_total_parameters, len_example_url )
+        
+    
     def GetURLBooleans( self ):
         
         return ( self._match_subdomains, self._keep_matched_subdomains, self._alphabetise_get_parameters, self._can_produce_multiple_files, self._should_be_associated_with_files, self._keep_fragment )

hydrus/core/HydrusConstants.py

@@ -70,7 +70,7 @@ options = {}
 # Misc
 
 NETWORK_VERSION = 19
-SOFTWARE_VERSION = 429
+SOFTWARE_VERSION = 430
 CLIENT_API_VERSION = 15
 
 SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )

hydrus/core/HydrusDB.py

@@ -45,7 +45,7 @@ def CheckCanVacuumCursor( db_path, c, stop_time = None ):
             
         
     
-    ( db_dir, db_filename ) = os.path.split( db_path )
+    db_dir = os.path.dirname( db_path )
     
     HydrusPaths.CheckHasSpaceForDBTransaction( db_dir, vacuum_estimate )
     
@@ -102,9 +102,11 @@ def ReadLargeIdQueryInSeparateChunks( cursor, select_statement, chunk_size ):
         
         chunk = [ temp_id for ( temp_id, ) in cursor.execute( 'SELECT temp_id FROM ' + table_name + ' WHERE job_id BETWEEN ? AND ?;', ( i, i + chunk_size - 1 ) ) ]
         
-        yield chunk
+        i += len( chunk )
         
-        i += chunk_size
+        num_done = i + 1
+        
+        yield ( chunk, num_done, num_to_do )
         
     
     cursor.execute( 'DROP TABLE ' + table_name + ';' )
@@ -145,13 +147,121 @@ def VacuumDB( db_path ):
     
     c.execute( 'PRAGMA journal_mode = {};'.format( HG.db_journal_mode ) )
     
+class DBCursorTransactionWrapper( object ):
+    
+    def __init__( self, c: sqlite3.Cursor, transaction_commit_period: int ):
+        
+        self._c = c
+        
+        self._transaction_commit_period = transaction_commit_period
+        
+        self._transaction_start_time = 0
+        self._in_transaction = False
+        self._transaction_contains_writes = False
+        
+        self._last_mem_refresh_time = HydrusData.GetNow()
+        self._last_wal_checkpoint_time = HydrusData.GetNow()
+        
+    
+    def BeginImmediate( self ):
+        
+        if not self._in_transaction:
+            
+            self._c.execute( 'BEGIN IMMEDIATE;' )
+            self._c.execute( 'SAVEPOINT hydrus_savepoint;' )
+            
+            self._transaction_start_time = HydrusData.GetNow()
+            self._in_transaction = True
+            self._transaction_contains_writes = False
+            
+        
+    
+    def Commit( self ):
+        
+        if self._in_transaction:
+            
+            self._c.execute( 'COMMIT;' )
+            
+            self._in_transaction = False
+            self._transaction_contains_writes = False
+            
+            if HG.db_journal_mode == 'WAL' and HydrusData.TimeHasPassed( self._last_wal_checkpoint_time + 1800 ):
+                
+                self._c.execute( 'PRAGMA wal_checkpoint(PASSIVE);' )
+                
+                self._last_wal_checkpoint_time = HydrusData.GetNow()
+                
+            
+            if HydrusData.TimeHasPassed( self._last_mem_refresh_time + 600 ):
+                
+                self._c.execute( 'DETACH mem;' )
+                self._c.execute( 'ATTACH ":memory:" AS mem;' )
+                
+                TemporaryIntegerTableNameCache.instance().Clear()
+                
+                self._last_mem_refresh_time = HydrusData.GetNow()
+                
+            
+        else:
+            
+            HydrusData.Print( 'Received a call to commit, but was not in a transaction!' )
+            
+        
+    
+    def CommitAndBegin( self ):
+        
+        if self._in_transaction:
+            
+            self.Commit()
+            
+            self.BeginImmediate()
+            
+        
+    
+    def InTransaction( self ):
+        
+        return self._in_transaction
+        
+    
+    def NotifyWriteOccuring( self ):
+        
+        self._transaction_contains_writes = True
+        
+    
+    def Rollback( self ):
+        
+        if self._in_transaction:
+            
+            self._c.execute( 'ROLLBACK TO hydrus_savepoint;' )
+            
+            # still in transaction
+            # transaction may no longer contain writes, but it isn't important to figure out that it doesn't
+            
+        else:
+            
+            HydrusData.Print( 'Received a call to rollback, but was not in a transaction!' )
+            
+        
+    
+    def Save( self ):
+        
+        self._c.execute( 'RELEASE hydrus_savepoint;' )
+        
+        self._c.execute( 'SAVEPOINT hydrus_savepoint;' )
+        
+    
+    def TimeToCommit( self ):
+        
+        return self._in_transaction and self._transaction_contains_writes and HydrusData.TimeHasPassed( self._transaction_start_time + self._transaction_commit_period )
+        
+    
 class HydrusDB( object ):
     
+    TRANSACTION_COMMIT_PERIOD = 30
+    
     READ_WRITE_ACTIONS = []
     UPDATE_WAIT = 2
     
-    TRANSACTION_COMMIT_TIME = 30
-    
     def __init__( self, controller, db_dir, db_name ):
         
         if HydrusPaths.GetFreeSpace( db_dir ) < 500 * 1048576:
@@ -167,19 +277,12 @@ class HydrusDB( object ):
         
         TemporaryIntegerTableNameCache()
         
-        self._transaction_started = 0
-        self._in_transaction = False
-        self._transaction_contains_writes = False
-        
         self._ssl_cert_filename = '{}.crt'.format( self._db_name )
         self._ssl_key_filename = '{}.key'.format( self._db_name )
         
         self._ssl_cert_path = os.path.join( self._db_dir, self._ssl_cert_filename )
         self._ssl_key_path = os.path.join( self._db_dir, self._ssl_key_filename )
         
-        self._last_mem_refresh_time = HydrusData.GetNow()
-        self._last_wal_checkpoint_time = HydrusData.GetNow()
-        
         main_db_filename = db_name
         
         if not main_db_filename.endswith( '.db' ):
@@ -213,6 +316,8 @@ class HydrusDB( object ):
         self._db = None
         self._c = None
         
+        self._cursor_transaction_wrapper = None
+        
         if os.path.exists( os.path.join( self._db_dir, self._db_filenames[ 'main' ] ) ):
             
             # open and close to clean up in case last session didn't close well
@@ -248,7 +353,7 @@ class HydrusDB( object ):
             
             try:
                 
-                self._BeginImmediate()
+                self._cursor_transaction_wrapper.BeginImmediate()
                 
             except Exception as e:
                 
@@ -259,7 +364,7 @@ class HydrusDB( object ):
                 
                 self._UpdateDB( version )
                 
-                self._Commit()
+                self._cursor_transaction_wrapper.Commit()
                 
                 self._is_db_updated = True
                 
@@ -269,7 +374,7 @@ class HydrusDB( object ):
                 
                 try:
                     
-                    self._Rollback()
+                    self._cursor_transaction_wrapper.Rollback()
                     
                 except Exception as rollback_e:
                     
@@ -326,18 +431,6 @@ class HydrusDB( object ):
         self._c.execute( 'ATTACH ? AS durable_temp;', ( db_path, ) )
         
     
-    def _BeginImmediate( self ):
-        
-        if not self._in_transaction:
-            
-            self._c.execute( 'BEGIN IMMEDIATE;' )
-            self._c.execute( 'SAVEPOINT hydrus_savepoint;' )
-            
-            self._transaction_started = HydrusData.GetNow()
-            self._in_transaction = True
-            
-        
-    
     def _CleanAfterJobWork( self ):
         
         self._pubsubs = []
@@ -349,9 +442,9 @@ class HydrusDB( object ):
         
         if self._db is not None:
             
-            if self._in_transaction:
+            if self._cursor_transaction_wrapper.InTransaction():
                 
-                self._Commit()
+                self._cursor_transaction_wrapper.Commit()
                 
             
             self._c.close()
@@ -363,41 +456,12 @@ class HydrusDB( object ):
             self._db = None
             self._c = None
             
+            self._cursor_transaction_wrapper = None
+            
             self._UnloadModules()
             
         
     
-    def _Commit( self ):
-        
-        if self._in_transaction:
-            
-            self._c.execute( 'COMMIT;' )
-            
-            self._in_transaction = False
-            
-            if HG.db_journal_mode == 'WAL' and HydrusData.TimeHasPassed( self._last_wal_checkpoint_time + 1800 ):
-                
-                self._c.execute( 'PRAGMA wal_checkpoint(PASSIVE);' )
-                
-                self._last_wal_checkpoint_time = HydrusData.GetNow()
-                
-            
-            if HydrusData.TimeHasPassed( self._last_mem_refresh_time + 600 ):
-                
-                self._c.execute( 'DETACH mem;' )
-                self._c.execute( 'ATTACH ":memory:" AS mem;' )
-                
-                TemporaryIntegerTableNameCache.instance().Clear()
-                
-                self._last_mem_refresh_time = HydrusData.GetNow()
-                
-            
-        else:
-            
-            HydrusData.Print( 'Received a call to commit, but was not in a transaction!' )
-            
-        
-    
     def _CreateDB( self ):
         
         raise NotImplementedError()
@@ -535,9 +599,7 @@ class HydrusDB( object ):
             
             self._CreateDB()
             
-            self._Commit()
-            
-            self._BeginImmediate()
+            self._cursor_transaction_wrapper.CommitAndBegin()
             
         
     
@@ -547,16 +609,14 @@ class HydrusDB( object ):
         
         db_path = os.path.join( self._db_dir, self._db_filenames[ 'main' ] )
         
-        db_just_created = not os.path.exists( db_path )
-        
         try:
             
             self._db = sqlite3.connect( db_path, isolation_level = None, detect_types = sqlite3.PARSE_DECLTYPES )
             
-            self._last_mem_refresh_time = HydrusData.GetNow()
-            
             self._c = self._db.cursor()
             
+            self._cursor_transaction_wrapper = DBCursorTransactionWrapper( self._c, self.TRANSACTION_COMMIT_PERIOD )
+            
             self._LoadModules()
             
             if HG.no_db_temp_files:
@@ -573,8 +633,6 @@ class HydrusDB( object ):
             raise HydrusExceptions.DBAccessException( 'Could not connect to database! This could be an issue related to WAL and network storage, or something else. If it is not obvious to you, please let hydrus dev know. Error follows:' + os.linesep * 2 + str( e ) )
             
         
-        self._last_mem_refresh_time = HydrusData.GetNow()
-        
         TemporaryIntegerTableNameCache.instance().Clear()
         
         # durable_temp is not excluded here
@@ -615,7 +673,7 @@ class HydrusDB( object ):
         
         try:
             
-            self._BeginImmediate()
+            self._cursor_transaction_wrapper.BeginImmediate()
             
         except Exception as e:
             
@@ -650,7 +708,7 @@ class HydrusDB( object ):
                 
                 self._current_status = 'db write locked'
                 
-                self._transaction_contains_writes = True
+                self._cursor_transaction_wrapper.NotifyWriteOccuring()
                 
             else:
                 
@@ -668,21 +726,17 @@ class HydrusDB( object ):
                 result = self._Write( action, *args, **kwargs )
                 
             
-            if self._transaction_contains_writes and HydrusData.TimeHasPassed( self._transaction_started + self.TRANSACTION_COMMIT_TIME ):
+            if self._cursor_transaction_wrapper.TimeToCommit():
                 
                 self._current_status = 'db committing'
                 
                 self.publish_status_update()
                 
-                self._Commit()
-                
-                self._BeginImmediate()
-                
-                self._transaction_contains_writes = False
+                self._cursor_transaction_wrapper.CommitAndBegin()
                 
             else:
                 
-                self._Save()
+                self._cursor_transaction_wrapper.Save()
                 
             
             self._DoAfterJobWork()
@@ -698,14 +752,12 @@ class HydrusDB( object ):
             
             try:
                 
-                self._Rollback()
+                self._cursor_transaction_wrapper.Rollback()
                 
             except Exception as rollback_e:
                 
                 HydrusData.Print( 'When the transaction failed, attempting to rollback the database failed. Please restart the client as soon as is convenient.' )
                 
-                self._in_transaction = False
-                
                 self._CloseDBCursor()
                 
                 self._InitDBCursor()
@@ -748,25 +800,6 @@ class HydrusDB( object ):
         HydrusData.Print( text )
         
     
-    def _Rollback( self ):
-        
-        if self._in_transaction:
-            
-            self._c.execute( 'ROLLBACK TO hydrus_savepoint;' )
-            
-        else:
-            
-            HydrusData.Print( 'Received a call to rollback, but was not in a transaction!' )
-            
-        
-    
-    def _Save( self ):
-        
-        self._c.execute( 'RELEASE hydrus_savepoint;' )
-        
-        self._c.execute( 'SAVEPOINT hydrus_savepoint;' )
-        
-    
     def _ShrinkMemory( self ):
         
         self._c.execute( 'PRAGMA shrink_memory;' )
@@ -995,13 +1028,9 @@ class HydrusDB( object ):
                 
             except queue.Empty:
                 
-                if self._transaction_contains_writes and HydrusData.TimeHasPassed( self._transaction_started + self.TRANSACTION_COMMIT_TIME ):
+                if self._cursor_transaction_wrapper.TimeToCommit():
                     
-                    self._Commit()
-                    
-                    self._BeginImmediate()
-                    
-                    self._transaction_contains_writes = False
+                    self._cursor_transaction_wrapper.CommitAndBegin()
                     
                 
             

hydrus/server/ServerDB.py

@@ -81,7 +81,7 @@ class DB( HydrusDB.HydrusDB ):
     
     READ_WRITE_ACTIONS = [ 'access_key', 'immediate_content_update', 'registration_keys' ]
     
-    TRANSACTION_COMMIT_TIME = 120
+    TRANSACTION_COMMIT_PERIOD = 120
     
     def __init__( self, controller, db_dir, db_name ):
         
@@ -3230,8 +3230,7 @@ class DB( HydrusDB.HydrusDB ):
                     
                     if HydrusData.TimeHasPassed( next_commit ):
                         
-                        self._Commit()
-                        self._BeginImmediate()
+                        self._cursor_transaction_wrapper.CommitAndBegin()
                         
                         next_commit = HydrusData.GetNow() + 60