Version 432

help/changelog.html

@@ -8,6 +8,42 @@
 		<div class="content">
 			<h3 id="changelog"><a href="#changelog">changelog</a></h3>
 			<ul>
+				<li><h3 id="version_432"><a href="#version_432">version 432</a></h3></li>
+				<ul>
+					<li>tag sorting:</li>
+					<li>the tag sort dropdown has been replaced with a dynamic control. rather than one big list with all possible permutations, you now work on each variable (sort type, asc/desc, group by) separately. what you are actually sorting is easier to understand and select</li>
+					<li>my stupid "lexicographic/incidence" labelling is replaced with the simpler and neater 'tag', 'subtag', and 'count'</li>
+					<li>when in the manage tags dialog and sorting by tag or subtag, you can now turn off the 'use sibling' sort.</li>
+					<li>I'd like to further neaten the workflow here, making the individual dropdowns flip back and forth with a mouse scroll in either directior rather than being just up/down allowed. let me know overall how you find this new control</li>
+					<li>the 'tag sort' object is updated behind the scenes as well. your old value should be converted automatically</li>
+					<li>fixed an issue with count tag sorting where deleted tag counts were being counted even when not displayed</li>
+					<li>if you try to search tags on a page of thumbnails that holds an invalid tag, this is now caught gracefully and you get a little popup saying 'please run the repair invalid tags routine'</li>
+					<li>.</li>
+					<li>misc:</li>
+					<li>the client now gives a once-per-boot warning popup if your session size exceeds 500k. for those who cannot reduce session size conveniently, this popup can be turned off under _options->gui pages_</li>
+					<li>when the file import options prohibit a file due to filesize or resolution etc.., it should now always record that as an 'ignored' result rather than an 'error'</li>
+					<li>fixed an unusual error popup in thread watcher display that could occur during session load. this problem seems to have been around for a long time, but it required a watcher in a previously saved and still valid 'wait a bit' error state and was only vulnerable for a few milliseconds, so it hadn't come up before. in any case, it is fixed</li>
+					<li>subscriptions with small 'first run' file limits now work better: if you create a subscription with a fairly small 'first run' file limit (this typically matters when the number is smaller than one of the site's gallery page's worth of results), subsequent normal checks with larger file limits will be more aggressive about noticing that they 'caught up' to that small initial sync (previously, they would sometimes incorrectly think the site just got some files tagged out of order and bump right past that initial 'already in db' batch and keep going until they hit their own file limit)</li>
+					<li>.</li>
+					<li>advanced string processing:</li>
+					<li>added a String Selector/Slicer object to the parsing system. this object allows you to select the nth item in a list of parsed strings or the mth to nth items. m can be 'start' and n can be 'end', and negative indices are allowed for both. pair it with the new Sorter for some neat new tricks!</li>
+					<li>the string processing edit UI is now _more_ multi-string-aware. the test panel has had a code cleanup pass and now has a list of all the starting strings in the test data (e.g. all the urls parsed by the formula that launched the UI) rather than just the first, and a list of all results from that list. selecting any of the starting strings populates the 'single string' area, so you can now zoom in on one particular string to see what is happening to it</li>
+					<li>the String Sorter edit UI now gets all the strings at that stage of processing, so you can review the sort properly</li>
+					<li>the new String Slicer edit UI similarly gets all the strings at that stage of processing</li>
+					<li>future updates will expand multi-string presentation and testing. I'd like to show the whole list at each stage</li>
+					<li>.</li>
+					<li>server/client api core improvements:</li>
+					<li>I had a go at supporting the Range header for file (basically this means anything non-html/json) requests. I added tests and it seems to work. as I understand this mostly applies to browsers pulling video from the Client API. to start, I am supporting single range requests. if it is needed, I'll try to get Multi Range requests right, but for now they'll 416</li>
+					<li>the client now understands 416 ("can't do that requested range m8") errors</li>
+					<li>I reworked the serverside error handling chain. this has been borked for a long time due to my own lack of understanding of twisted's deferred system, and certain late-stage errors were just not being handled right. the server should no longer hang on these and now should print error info correctly, including a rough 500 in true late emergencies, and terminate the connection correctly</li>
+					<li>.</li>
+					<li>boring:</li>
+					<li>fixed up a handful of typo-borked unit tests</li>
+					<li>fixed my ordinal (xst, xnd, xrd, xth) text generator to deal with 11, 12, and 13 correctly lmao</li>
+					<li>started some db maintenance routines and logistics to recover definitions and remove orphans in future, I feel great about it so far, but it'll have to wait for more of my db 'modules' refactoring to be more useful</li>
+					<li>updated the mpv dll on the Windows release to 2021-02-28, it may improve some video support/performance</li>
+					<li>updated sqlite dll on the Windows release to 3.34.1</li>
+				</ul>
 				<li><h3 id="version_431"><a href="#version_431">version 431</a></h3></li>
 				<ul>
 					<li>misc:</li>

hydrus/client/ClientConstants.py

@@ -274,17 +274,6 @@ SHUTDOWN_TIMESTAMP_VACUUM = 0
 SHUTDOWN_TIMESTAMP_FATTEN_AC_CACHE = 1
 SHUTDOWN_TIMESTAMP_DELETE_ORPHANS = 2
 
-SORT_BY_LEXICOGRAPHIC_ASC = 8
-SORT_BY_LEXICOGRAPHIC_DESC = 9
-SORT_BY_INCIDENCE_ASC = 10
-SORT_BY_INCIDENCE_DESC = 11
-SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC = 12
-SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC = 13
-SORT_BY_INCIDENCE_NAMESPACE_ASC = 14
-SORT_BY_INCIDENCE_NAMESPACE_DESC = 15
-SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC = 16
-SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC = 17
-
 SORT_FILES_BY_FILESIZE = 0
 SORT_FILES_BY_DURATION = 1
 SORT_FILES_BY_IMPORT_TIME = 2

hydrus/client/ClientDefaults.py

@@ -74,7 +74,6 @@ def GetClientDefaultOptions():
     options[ 'confirm_client_exit' ] = False
     
     options[ 'default_tag_repository' ] = CC.DEFAULT_LOCAL_TAG_SERVICE_KEY
-    options[ 'default_tag_sort' ] = CC.SORT_BY_LEXICOGRAPHIC_ASC
     
     options[ 'pause_export_folders_sync' ] = False
     options[ 'pause_import_folders_sync' ] = False

hydrus/client/ClientExporting.py

@@ -644,7 +644,9 @@ class SidecarExporter( HydrusSerialisable.SerialisableBase ):
             
             all_tags = list( all_tags )
             
-            ClientTagSorting.SortTags( CC.SORT_BY_LEXICOGRAPHIC_DESC, all_tags )
+            tag_sort = ClientTagSorting.TagSort.STATICGetTextASCDefault()
+            
+            ClientTagSorting.SortTags( tag_sort, all_tags )
             
             txt_path = os.path.join( directory, filename + '.txt' )
             

hydrus/client/ClientOptions.py

@@ -232,6 +232,8 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
         self._dictionary[ 'booleans' ][ 'expand_parents_on_storage_taglists' ] = True
         self._dictionary[ 'booleans' ][ 'expand_parents_on_storage_autocomplete_taglists' ] = True
         
+        self._dictionary[ 'booleans' ][ 'show_session_size_warnings' ] = True
+        
         #
         
         self._dictionary[ 'colours' ] = HydrusSerialisable.SerialisableDictionary()
@@ -612,6 +614,12 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
         
         self._dictionary[ 'default_collect' ] = ClientMedia.MediaCollect()
         
+        #
+        
+        from hydrus.client.metadata import ClientTagSorting
+        
+        self._dictionary[ 'default_tag_sort' ] = ClientTagSorting.TagSort.STATICGetTextASCDefault()
+        
     
     def _InitialiseFromSerialisableInfo( self, serialisable_info ):
         
@@ -897,6 +905,14 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
             
         
     
+    def GetDefaultTagSort( self ):
+        
+        with self._lock:
+            
+            return self._dictionary[ 'default_tag_sort' ]
+            
+        
+    
     def GetDefaultWatcherCheckerOptions( self ):
         
         with self._lock:
@@ -1285,6 +1301,14 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
             
         
     
+    def SetDefaultTagSort( self, tag_sort ):
+        
+        with self._lock:
+            
+            self._dictionary[ 'default_tag_sort' ] = tag_sort
+            
+        
+    
     def SetDefaultWatcherCheckerOptions( self, checker_options ):
         
         with self._lock:

hydrus/client/ClientParsing.py

@@ -3710,6 +3710,165 @@ class StringMatch( StringProcessingStep ):
     
 HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_STRING_MATCH ] = StringMatch
 
+class StringSlicer( StringProcessingStep ):
+    
+    SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_STRING_SLICER
+    SERIALISABLE_NAME = 'String Selector/Slicer'
+    SERIALISABLE_VERSION = 1
+    
+    def __init__( self, index_start: typing.Optional[ int ] = None, index_end: typing.Optional[ int ] = None ):
+        
+        StringProcessingStep.__init__( self )
+        
+        self._index_start = index_start
+        self._index_end = index_end
+        
+    
+    def _GetSerialisableInfo( self ):
+        
+        return ( self._index_start, self._index_end )
+        
+    
+    def _InitialiseFromSerialisableInfo( self, serialisable_info ):
+        
+        ( self._index_start, self._index_end ) = serialisable_info
+        
+    
+    def GetIndexStartEnd( self ) -> typing.Tuple[ typing.Optional[ int ], typing.Optional[ int ] ]:
+        
+        return ( self._index_start, self._index_end )
+        
+    
+    def MakesChanges( self ) -> bool:
+        
+        return self._index_start is not None or self._index_end is not None
+        
+    
+    def SelectsNothingEver( self ) -> bool:
+        
+        if self._index_end == 0:
+            
+            return True
+            
+        
+        if self._index_start is None or self._index_end is None:
+            
+            return False
+            
+        
+        both_positive = self._index_start >= 0 and self._index_end >= 0
+        both_negative = self._index_start < 0 and self._index_end < 0
+        
+        if both_positive or both_negative:
+            
+            if self._index_start >= self._index_end:
+                
+                return True
+                
+            
+        
+        return False
+        
+    
+    def SelectsOne( self ) -> bool:
+        
+        if self.SelectsNothingEver():
+            
+            return False
+            
+        
+        if self._index_start == -1 and self._index_end is None:
+            
+            return True
+            
+        
+        if self._index_start is None or self._index_end is None:
+            
+            return False
+            
+        
+        both_positive = self._index_start >= 0 and self._index_end >= 0
+        both_negative = self._index_start < 0 and self._index_end < 0
+        
+        return ( both_positive or both_negative ) and self._index_start == self._index_end - 1
+        
+    
+    def Slice( self, texts: typing.Sequence[ str ] ) -> typing.List[ str ]:
+        
+        try:
+            
+            if self._index_start is None and self._index_end is None:
+                
+                return list( texts )
+                
+            elif self._index_end is None:
+                
+                return texts[ self._index_start : ]
+                
+            elif self._index_start is None:
+                
+                return texts[ : self._index_end ]
+                
+            else:
+                
+                return texts[ self._index_start : self._index_end ]
+                
+            
+        except IndexError as e:
+            
+            return []
+            
+        
+    
+    def ToString( self, simple = False, with_type = False ) -> str:
+        
+        if simple:
+            
+            return 'selector/slicer'
+            
+        
+        if self.SelectsNothingEver():
+            
+            result = 'selecting nothing'
+            
+        elif self.SelectsOne():
+            
+            result = 'selecting the {} string'.format( HydrusData.ConvertIndexToPrettyOrdinalString( self._index_start ) )
+            
+        elif self._index_start is None and self._index_end is None:
+            
+            result = 'selecting everything'
+            
+        elif self._index_end is None:
+            
+            result = 'selecting the {} string and onwards'.format( HydrusData.ConvertIndexToPrettyOrdinalString( self._index_start ) )
+            
+        elif self._index_start is None:
+            
+            result = 'selecting up to and including the {} string'.format( HydrusData.ConvertIndexToPrettyOrdinalString( self._index_end - 1 ) )
+            
+        else:
+            
+            result = 'selecting the {} string up to and including the {} string'.format( HydrusData.ConvertIndexToPrettyOrdinalString( self._index_start ), HydrusData.ConvertIndexToPrettyOrdinalString( self._index_end - 1 ) )
+            
+        
+        if with_type:
+            
+            if self.SelectsOne():
+                
+                result = 'SELECT: {}'.format( result )
+                
+            else:
+                
+                result = 'SLICE: {}'.format( result )
+                
+            
+        
+        return result
+        
+    
+HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_STRING_SLICER ] = StringSlicer
+
 sort_str_enum = {
     CONTENT_PARSER_SORT_TYPE_NONE : 'no sorting',
     CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC : 'strict lexicographic',
@@ -3974,7 +4133,7 @@ class StringProcessor( HydrusSerialisable.SerialisableBase ):
         return proc_strings
         
     
-    def ProcessStrings( self, starting_strings: typing.Iterable[ str ], max_steps_allowed = None ) -> typing.List[ str ]:
+    def ProcessStrings( self, starting_strings: typing.Iterable[ str ], max_steps_allowed = None, no_slicing = False ) -> typing.List[ str ]:
         
         current_strings = list( starting_strings )
         
@@ -3996,6 +4155,24 @@ class StringProcessor( HydrusSerialisable.SerialisableBase ):
                     next_strings = current_strings
                     
                 
+            elif isinstance( processing_step, StringSlicer ):
+                
+                if no_slicing:
+                    
+                    next_strings = current_strings
+                    
+                else:
+                    
+                    try:
+                        
+                        next_strings = processing_step.Slice( current_strings )
+                        
+                    except:
+                        
+                        next_strings = current_strings
+                        
+                    
+                
             else:
                 
                 next_strings = []
@@ -4096,6 +4273,11 @@ class StringProcessor( HydrusSerialisable.SerialisableBase ):
                 components.append( 'sorting' )
                 
             
+            if True in ( isinstance( ps, StringSlicer ) for ps in self._processing_steps ):
+                
+                components.append( 'selecting/slicing' )
+                
+            
             return 'some {}'.format( ', '.join( components ) )
             
         

hydrus/client/db/ClientDB.py

@@ -10736,9 +10736,25 @@ class DB( HydrusDB.HydrusDB ):
         
         tag_ids_to_full_counts = {}
         
+        showed_bad_tag_error = False
+        
         for ( i, ( tag, ( current_count, pending_count ) ) ) in enumerate( tags_to_counts.items() ):
             
-            tag_id = self.modules_tags.GetTagId( tag )
+            try:
+                
+                tag_id = self.modules_tags.GetTagId( tag )
+                
+            except HydrusExceptions.TagSizeException:
+                
+                if not showed_bad_tag_error:
+                    
+                    showed_bad_tag_error = True
+                    
+                    HydrusData.ShowText( 'Hey, you seem to have an invalid tag in view right now! Please run the \'repair invalid tags\' routine under the \'database\' menu asap!' )
+                    
+                
+                continue
+                
             
             tag_ids_to_full_counts[ tag_id ] = ( current_count, max_current_count, pending_count, max_pending_count )
             
@@ -15108,6 +15124,97 @@ class DB( HydrusDB.HydrusDB ):
         return result
         
     
+    def _RecoverFromMissingDefinitions( self, content_type ):
+        
+        # this is not finished, but basics are there
+        # remember this func uses a bunch of similar tech for the eventual orphan definition cleansing routine
+        # we just have to extend modules functionality to cover all content tables and we are good to go
+        
+        if content_type == HC.CONTENT_TYPE_HASH:
+            
+            definition_column_name = 'hash_id'
+            
+        
+        # eventually migrate this gubbins to cancellable async done in parts, which means generating, handling, and releasing the temp table name more cleverly
+        
+        # job presentation to UI
+        
+        all_tables_and_columns = []
+        
+        for module in self._modules:
+            
+            all_tables_and_columns.extend( module.GetTablesAndColumnsThatUseDefinitions( HC.CONTENT_TYPE_HASH ) )
+            
+        
+        temp_all_useful_definition_ids_table_name = 'durable_temp.all_useful_definition_ids_{}'.format( os.urandom( 8 ).hex() )
+        
+        self._c.execute( 'CREATE TABLE {} ( {} INTEGER PRIMARY KEY );'.format( temp_all_useful_definition_ids_table_name, definition_column_name ) )
+        
+        try:
+            
+            num_to_do = 0
+            
+            for ( table_name, column_name ) in all_tables_and_columns:
+                
+                query = 'INSERT OR IGNORE INTO {} ( {} ) SELECT DISTINCT {} FROM {};'.format(
+                    temp_all_useful_definition_ids_table_name,
+                    definition_column_name,
+                    column_name,
+                    table_name
+                )
+                
+                self._c.execute( query )
+                
+                num_to_do += self._GetRowCount()
+                
+            
+            num_missing = 0
+            num_recovered = 0
+            
+            batch_of_definition_ids = self._c.execute( 'SELECT {} FROM {} LIMIT 1024;'.format( definition_column_name, temp_all_useful_definition_ids_table_name ) )
+            
+            while len( batch_of_definition_ids ) > 1024:
+                
+                for definition_id in batch_of_definition_ids:
+                    
+                    if not self.modules_hashes.HasHashId( definition_id ):
+                        
+                        if content_type == HC.CONTENT_TYPE_HASH and self.modules_hashes_local_cache.HasHashId( definition_id ):
+                            
+                            hash = self.modules_hashes_local_cache.GetHash( definition_id )
+                            
+                            self._c.execute( 'INSERT OR IGNORE INTO hashes ( hash_id, hash ) VALUES ( ?, ? );', ( definition_id, sqlite3.Binary( hash ) ) )
+                            
+                            HydrusData.Print( '{} {} had no master definition, but I was able to recover from the local cache'.format( definition_column_name, definition_id ) )
+                            
+                            num_recovered += 1
+                            
+                        else:
+                            
+                            HydrusData.Print( '{} {} had no master definition, it has been purged from the database!'.format( definition_column_name, definition_id ) )
+                            
+                            for ( table_name, column_name ) in all_tables_and_columns:
+                                
+                                self._c.execute( 'DELETE FROM {} WHERE {} = ?;'.format( table_name, column_name ), ( definition_id, ) )
+                                
+                            
+                            # tell user they will want to run clear orphan files, reset service cache info, and may need to recalc some autocomplete counts depending on total missing definitions
+                            # I should clear service info based on content_type
+                            
+                            num_missing += 1
+                            
+                        
+                    
+                
+                batch_of_definition_ids = self._c.execute( 'SELECT {} FROM {} LIMIT 1024;'.format( definition_column_name, temp_all_useful_definition_ids_table_name ) )
+                
+            
+        finally:
+            
+            self._c.execute( 'DROP TABLE {};'.format( temp_all_useful_definition_ids_table_name ) )
+            
+        
+    
     def _RegenerateLocalHashCache( self ):
         
         job_key = ClientThreading.JobKey( cancellable = True )
@@ -19200,6 +19307,85 @@ class DB( HydrusDB.HydrusDB ):
                 
             
         
+        if version == 431:
+            
+            try:
+                
+                new_options = self.modules_serialisable.GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_CLIENT_OPTIONS )
+                
+                old_options = self._GetOptions()
+                
+                SORT_BY_LEXICOGRAPHIC_ASC = 8
+                SORT_BY_LEXICOGRAPHIC_DESC = 9
+                SORT_BY_INCIDENCE_ASC = 10
+                SORT_BY_INCIDENCE_DESC = 11
+                SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC = 12
+                SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC = 13
+                SORT_BY_INCIDENCE_NAMESPACE_ASC = 14
+                SORT_BY_INCIDENCE_NAMESPACE_DESC = 15
+                SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC = 16
+                SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC = 17
+                
+                old_default_tag_sort = old_options[ 'default_tag_sort' ]
+                
+                from hydrus.client.metadata import ClientTagSorting
+                
+                sort_type = ClientTagSorting.SORT_BY_HUMAN_TAG
+                
+                if old_default_tag_sort in ( SORT_BY_LEXICOGRAPHIC_ASC, SORT_BY_LEXICOGRAPHIC_DESC, SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC ):
+                    
+                    sort_type = ClientTagSorting.SORT_BY_HUMAN_TAG
+                    
+                elif old_default_tag_sort in ( SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC, SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC ):
+                    
+                    sort_type = ClientTagSorting.SORT_BY_HUMAN_SUBTAG
+                    
+                elif old_default_tag_sort in ( SORT_BY_INCIDENCE_ASC, SORT_BY_INCIDENCE_DESC, SORT_BY_INCIDENCE_NAMESPACE_ASC, SORT_BY_INCIDENCE_NAMESPACE_DESC ):
+                    
+                    sort_type = ClientTagSorting.SORT_BY_COUNT
+                    
+                
+                if old_default_tag_sort in ( SORT_BY_INCIDENCE_ASC, SORT_BY_INCIDENCE_NAMESPACE_ASC, SORT_BY_LEXICOGRAPHIC_ASC, SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC, SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC ):
+                    
+                    sort_order = CC.SORT_ASC
+                    
+                else:
+                    
+                    sort_order = CC.SORT_DESC
+                    
+                
+                use_siblings = True
+                
+                if old_default_tag_sort in ( SORT_BY_INCIDENCE_NAMESPACE_ASC, SORT_BY_INCIDENCE_NAMESPACE_DESC, SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
+                    
+                    group_by = ClientTagSorting.GROUP_BY_NAMESPACE
+                    
+                else:
+                    
+                    group_by = ClientTagSorting.GROUP_BY_NOTHING
+                    
+                
+                tag_sort = ClientTagSorting.TagSort(
+                    sort_type = sort_type,
+                    sort_order = sort_order,
+                    use_siblings = use_siblings,
+                    group_by = group_by
+                )
+                
+                new_options.SetDefaultTagSort( tag_sort )
+                
+                self.modules_serialisable.SetJSONDump( new_options )
+                
+            except Exception as e:
+                
+                HydrusData.PrintException( e )
+                
+                message = 'Trying to convert your old default tag sort to the new format failed! Please set it again in the options.'
+                
+                self.pub_initial_message( message )
+                
+            
+        
         self._controller.frame_splash_status.SetTitleText( 'updated db to v{}'.format( HydrusData.ToHumanInt( version + 1 ) ) )
         
         self._c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )

hydrus/client/db/ClientDBDefinitionsCache.py

@@ -182,6 +182,20 @@ class ClientDBCacheLocalHashes( HydrusDBModule.HydrusDBModule ):
         return hash_ids_to_hashes
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        # we actually provide a backup, which we may want to automate later in mappings caches etc...
+        
+        return []
+        
+    
+    def HasHashId( self, hash_id: int ):
+        
+        result = self._c.execute( 'SELECT 1 FROM local_hashes_cache WHERE hash_id = ?;', ( hash_id, ) ).fetchone()
+        
+        return result is not None
+        
+    
 class ClientDBCacheLocalTags( HydrusDBModule.HydrusDBModule ):
     
     def __init__( self, cursor: sqlite3.Cursor, modules_tags: ClientDBMaster.ClientDBMasterTags ):
@@ -276,6 +290,13 @@ class ClientDBCacheLocalTags( HydrusDBModule.HydrusDBModule ):
         return expected_table_names
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        # we actually provide a backup, which we may want to automate later in mappings caches etc...
+        
+        return []
+        
+    
     def GetTag( self, tag_id ) -> str:
         
         self._PopulateTagIdsToTagsCache( ( tag_id, ) )

hydrus/client/db/ClientDBFilesMetadataBasic.py

@@ -129,6 +129,16 @@ class ClientDBFilesMetadataBasic( HydrusDBModule.HydrusDBModule ):
         return sum( ( 1 for mime in result if mime in HC.SEARCHABLE_MIMES ) )
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        if HC.CONTENT_TYPE_HASH:
+            
+            return [ ( 'files_info', 'hash_id' ) ]
+            
+        
+        return []
+        
+    
     def GetTotalSize( self, hash_ids: typing.Collection[ int ] ) -> int:
         
         if len( hash_ids ) == 1:

hydrus/client/db/ClientDBMappingsStorage.py

@@ -1,6 +1,7 @@
 import sqlite3
 import typing
 
+from hydrus.core import HydrusConstants as HC
 from hydrus.core import HydrusData
 from hydrus.core import HydrusDB
 from hydrus.core import HydrusDBModule
@@ -78,3 +79,45 @@ class ClientDBMappingsStorage( HydrusDBModule.HydrusDBModule ):
         self._CreateIndex( petitioned_mappings_table_name, [ 'hash_id', 'tag_id' ], unique = True )
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        if HC.CONTENT_TYPE_HASH:
+            
+            tables_and_columns = []
+            
+            for service_id in self.modules_services.GetServiceIds( HC.REAL_TAG_SERVICES ):
+                
+                ( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateMappingsTableNames( service_id )
+                
+                tables_and_columns.extend( [
+                    ( current_mappings_table_name, 'hash_id' ),
+                    ( deleted_mappings_table_name, 'hash_id' ),
+                    ( pending_mappings_table_name, 'hash_id' ),
+                    ( petitioned_mappings_table_name, 'hash_id' )
+                ] )
+                
+            
+            return tables_and_columns
+            
+        elif HC.CONTENT_TYPE_TAG:
+            
+            tables_and_columns = []
+            
+            for service_id in self.modules_services.GetServiceIds( HC.REAL_TAG_SERVICES ):
+                
+                ( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateMappingsTableNames( service_id )
+                
+                tables_and_columns.extend( [
+                    ( current_mappings_table_name, 'tag_id' ),
+                    ( deleted_mappings_table_name, 'tag_id' ),
+                    ( pending_mappings_table_name, 'tag_id' ),
+                    ( petitioned_mappings_table_name, 'tag_id' )
+                ] )
+                
+            
+            return tables_and_columns
+            
+        
+        return []
+        
+    

hydrus/client/db/ClientDBMaster.py

@@ -2,6 +2,7 @@ import os
 import sqlite3
 import typing
 
+from hydrus.core import HydrusConstants as HC
 from hydrus.core import HydrusData
 from hydrus.core import HydrusDB
 from hydrus.core import HydrusDBModule
@@ -282,6 +283,16 @@ class ClientDBMasterHashes( HydrusDBModule.HydrusDBModule ):
         return hash_ids_to_hashes
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        if HC.CONTENT_TYPE_HASH:
+            
+            return [ ( 'local_hashes', 'hash_id' ) ]
+            
+        
+        return []
+        
+    
     def HasExtraHashes( self, hash_id ):
         
         result = self._c.execute( 'SELECT 1 FROM local_hashes WHERE hash_id = ?;', ( hash_id, ) ).fetchone()
@@ -289,6 +300,13 @@ class ClientDBMasterHashes( HydrusDBModule.HydrusDBModule ):
         return result is not None
         
     
+    def HasHashId( self, hash_id: int ):
+        
+        result = self._c.execute( 'SELECT 1 FROM hashes WHERE hash_id = ?;', ( hash_id, ) ).fetchone()
+        
+        return result is not None
+        
+    
     def SetExtraHashes( self, hash_id, md5, sha1, sha512 ):
         
         self._c.execute( 'INSERT OR IGNORE INTO local_hashes ( hash_id, md5, sha1, sha512 ) VALUES ( ?, ?, ?, ? );', ( hash_id, sqlite3.Binary( md5 ), sqlite3.Binary( sha1 ), sqlite3.Binary( sha512 ) ) )
@@ -346,6 +364,11 @@ class ClientDBMasterTexts( HydrusDBModule.HydrusDBModule ):
         return label_id
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        return []
+        
+    
     def GetText( self, text_id ):
         
         result = self._c.execute( 'SELECT text FROM texts WHERE text_id = ?;', ( text_id, ) ).fetchone()
@@ -522,6 +545,13 @@ class ClientDBMasterTags( HydrusDBModule.HydrusDBModule ):
         return subtag_id
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        # maybe content type subtag/namespace, which would useful for bad subtags, although that's tricky because then the knock-on is killing tag definition rows
+        
+        return []
+        
+    
     def GetTag( self, tag_id ) -> str:
         
         self._PopulateTagIdsToTagsCache( ( tag_id, ) )
@@ -539,6 +569,8 @@ class ClientDBMasterTags( HydrusDBModule.HydrusDBModule ):
             
         except HydrusExceptions.TagSizeException:
             
+            # update this to instead go 'hey, does the dirty tag exist?' if it does, run the fix invalid tags routine
+            
             raise HydrusExceptions.TagSizeException( '"{}" tag seems not valid--when cleaned, it ends up with zero size!'.format( tag ) )
             
         
@@ -716,6 +748,13 @@ class ClientDBMasterURLs( HydrusDBModule.HydrusDBModule ):
         return expected_table_names
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        # if content type is a domain, then give urls? bleh
+        
+        return []
+        
+    
     def GetURLDomainId( self, domain ):
         
         result = self._c.execute( 'SELECT domain_id FROM url_domains WHERE domain = ?;', ( domain, ) ).fetchone()

hydrus/client/db/ClientDBSerialisable.py

@@ -337,6 +337,11 @@ class ClientDBSerialisable( HydrusDBModule.HydrusDBModule ):
         return value
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        return []
+        
+    
     def GetYAMLDump( self, dump_type, dump_name = None ):
         
         if dump_name is None:

hydrus/client/db/ClientDBServices.py

@@ -178,6 +178,11 @@ class ClientDBMasterServices( HydrusDBModule.HydrusDBModule ):
         return set( self._service_keys_to_service_ids.keys() )
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        return []
+        
+    
     def UpdateService( self, service: ClientServices.Service ):
         
         ( service_key, service_type, name, dictionary ) = service.ToTuple()

hydrus/client/gui/ClientGUI.py

@@ -324,6 +324,8 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ):
         
         self._persistent_mpv_widgets = []
         
+        self._have_shown_session_size_warning = False
+        
         self._closed_pages = []
         
         self._lock = threading.Lock()
@@ -5644,6 +5646,13 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
         
         self._last_total_page_weight = total_active_weight + total_closed_weight
         
+        if total_active_weight > 500000 and self._controller.new_options.GetBoolean( 'show_session_size_warnings' ) and not self._have_shown_session_size_warning:
+            
+            self._have_shown_session_size_warning = True
+            
+            HydrusData.ShowText( 'Your session weight is {}, which is pretty big! To keep your UI lag-free and avoid potential session saving problems that occur around 2 million weight, please try to close some pages or clear some finished downloaders!'.format( HydrusData.ToHumanInt( total_active_weight ) ) )
+            
+        
         ClientGUIMenus.AppendMenuLabel( menu, '{} pages open'.format( HydrusData.ToHumanInt( total_active_page_count ) ), 'You have this many pages open.' )
         ClientGUIMenus.AppendMenuLabel( menu, 'total session weight: {}'.format( HydrusData.ToHumanInt( self._last_total_page_weight ) ), 'Your session is this heavy.' )
         

hydrus/client/gui/ClientGUICanvas.py

@@ -2001,7 +2001,9 @@ class CanvasWithDetails( Canvas ):
             
             tags_i_want_to_display = list( tags_i_want_to_display )
             
-            ClientTagSorting.SortTags( HC.options[ 'default_tag_sort' ], tags_i_want_to_display )
+            tag_sort = HG.client_controller.new_options.GetDefaultTagSort()
+            
+            ClientTagSorting.SortTags( tag_sort, tags_i_want_to_display )
             
             current_y = 3
             

hydrus/client/gui/ClientGUIScrolledPanelsManagement.py

@@ -30,6 +30,7 @@ from hydrus.client.gui import ClientGUIScrolledPanels
 from hydrus.client.gui import ClientGUIScrolledPanelsEdit
 from hydrus.client.gui import ClientGUIShortcuts
 from hydrus.client.gui import ClientGUIStyle
+from hydrus.client.gui import ClientGUITagSorting
 from hydrus.client.gui import ClientGUITime
 from hydrus.client.gui import ClientGUITopLevelWindowsPanels
 from hydrus.client.gui import QtPorting as QP
@@ -1389,6 +1390,10 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             self._number_of_gui_session_backups.setToolTip( 'The client keeps multiple rolling backups of your gui sessions. If you have very large sessions, you might like to reduce this number.' )
             
+            self._show_session_size_warnings = QW.QCheckBox( self._sessions_panel )
+            
+            self._show_session_size_warnings.setToolTip( 'This will give you a once-per-boot warning popup if your active session contains more than 500k objects.' )
+            
             #
             
             self._pages_panel = ClientGUICommon.StaticBox( self, 'pages' )
@@ -1473,6 +1478,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             self._number_of_gui_session_backups.setValue( self._new_options.GetInteger( 'number_of_gui_session_backups' ) )
             
+            self._show_session_size_warnings.setChecked( self._new_options.GetBoolean( 'show_session_size_warnings' ) )
+            
             self._default_new_page_goes.SetValue( self._new_options.GetInteger( 'default_new_page_goes' ) )
             
             self._notebook_tab_alignment.SetValue( self._new_options.GetInteger( 'notebook_tab_alignment' ) )
@@ -1507,6 +1514,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             rows.append( ( 'If \'last session\' above, autosave it how often (minutes)?', self._last_session_save_period_minutes ) )
             rows.append( ( 'If \'last session\' above, only autosave during idle time?', self._only_save_last_session_during_idle ) )
             rows.append( ( 'Number of session backups to keep: ', self._number_of_gui_session_backups ) )
+            rows.append( ( 'Show warning popup if session size exceeds 500k: ', self._show_session_size_warnings ) )
             
             sessions_gridbox = ClientGUICommon.WrapInGrid( self._sessions_panel, rows )
             
@@ -1582,6 +1590,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             self._new_options.SetInteger( 'number_of_gui_session_backups', self._number_of_gui_session_backups.value() )
             
+            self._new_options.SetBoolean( 'show_session_size_warnings', self._show_session_size_warnings.isChecked() )
+            
             self._new_options.SetBoolean( 'only_save_last_session_during_idle', self._only_save_last_session_during_idle.isChecked() )
             
             self._new_options.SetInteger( 'default_new_page_goes', self._default_new_page_goes.GetValue() )
@@ -2914,18 +2924,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             general_panel = ClientGUICommon.StaticBox( self, 'general tag options' )
             
-            self._default_tag_sort = ClientGUICommon.BetterChoice( general_panel )
-            
-            self._default_tag_sort.addItem( 'lexicographic (a-z)', CC.SORT_BY_LEXICOGRAPHIC_ASC )
-            self._default_tag_sort.addItem( 'lexicographic (z-a)', CC.SORT_BY_LEXICOGRAPHIC_DESC )
-            self._default_tag_sort.addItem( 'lexicographic (a-z) (group unnamespaced)', CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC )
-            self._default_tag_sort.addItem( 'lexicographic (z-a) (group unnamespaced)', CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC )
-            self._default_tag_sort.addItem( 'lexicographic (a-z) (ignore namespace)', CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC )
-            self._default_tag_sort.addItem( 'lexicographic (z-a) (ignore namespace)', CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC )
-            self._default_tag_sort.addItem( 'incidence (desc)', CC.SORT_BY_INCIDENCE_DESC )
-            self._default_tag_sort.addItem( 'incidence (asc)', CC.SORT_BY_INCIDENCE_ASC )
-            self._default_tag_sort.addItem( 'incidence (desc) (grouped by namespace)', CC.SORT_BY_INCIDENCE_NAMESPACE_DESC )
-            self._default_tag_sort.addItem( 'incidence (asc) (grouped by namespace)', CC.SORT_BY_INCIDENCE_NAMESPACE_ASC )
+            self._default_tag_sort = ClientGUITagSorting.TagSortControl( general_panel, self._new_options.GetDefaultTagSort(), show_siblings = True )
             
             self._default_tag_repository = ClientGUICommon.BetterChoice( general_panel )
             
@@ -2949,8 +2948,6 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             #
             
-            self._default_tag_sort.SetValue( HC.options[ 'default_tag_sort' ] )
-            
             self._default_tag_service_search_page.addItem( 'all known tags', CC.COMBINED_TAG_SERVICE_KEY )
             
             services = HG.client_controller.services_manager.GetServices( HC.REAL_TAG_SERVICES )
@@ -3017,7 +3014,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
         def UpdateOptions( self ):
             
             HC.options[ 'default_tag_repository' ] = self._default_tag_repository.GetValue()
-            HC.options[ 'default_tag_sort' ] = QP.GetClientData( self._default_tag_sort, self._default_tag_sort.currentIndex() )
+            self._new_options.SetDefaultTagSort( self._default_tag_sort.GetValue() )
             
             self._new_options.SetBoolean( 'expand_parents_on_storage_taglists', self._expand_parents_on_storage_taglists.isChecked() )
             self._new_options.SetBoolean( 'expand_parents_on_storage_autocomplete_taglists', self._expand_parents_on_storage_autocomplete_taglists.isChecked() )

hydrus/client/gui/ClientGUIStringPanels.py

@@ -1,6 +1,7 @@
 import re
 import typing
 
+from qtpy import QtCore as QC
 from qtpy import QtWidgets as QW
 
 from hydrus.core import HydrusConstants as HC
@@ -19,6 +20,285 @@ from hydrus.client.gui.lists import ClientGUIListBoxes
 from hydrus.client.gui.lists import ClientGUIListConstants as CGLC
 from hydrus.client.gui.lists import ClientGUIListCtrl
 
+NO_RESULTS_TEXT = 'no results'
+
+class MultilineStringConversionTestPanel( QW.QWidget ):
+    
+    textSelected = QC.Signal( str )
+    
+    def __init__( self, parent: QW.QWidget, string_processor: ClientParsing.StringProcessor ):
+        
+        QW.QWidget.__init__( self, parent )
+        
+        self._string_processor = string_processor
+        
+        self._test_data = QW.QListWidget( self )
+        
+        self._test_data.setSelectionMode( QW.QListWidget.SingleSelection )
+        
+        self._result_data = QW.QListWidget( self )
+        
+        self._result_data.setSelectionMode( QW.QListView.NoSelection )
+        
+        #
+        
+        left_vbox = QP.VBoxLayout()
+        right_vbox = QP.VBoxLayout()
+        
+        QP.AddToLayout( left_vbox, ClientGUICommon.BetterStaticText( self, label = 'starting strings' ), CC.FLAGS_CENTER )
+        QP.AddToLayout( left_vbox, self._test_data, CC.FLAGS_EXPAND_BOTH_WAYS )
+        QP.AddToLayout( right_vbox, ClientGUICommon.BetterStaticText( self, label = 'processed strings' ), CC.FLAGS_CENTER )
+        QP.AddToLayout( right_vbox, self._result_data, CC.FLAGS_EXPAND_BOTH_WAYS )
+        
+        hbox = QP.HBoxLayout()
+        
+        QP.AddToLayout( hbox, left_vbox, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
+        QP.AddToLayout( hbox, right_vbox, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
+        
+        self.setLayout( hbox )
+        
+        self._test_data.itemSelectionChanged.connect( self.EventSelection )
+        
+    
+    def _GetStartingTexts( self ):
+        
+        return [ self._test_data.item( i ).data( QC.Qt.UserRole ) for i in range( self._test_data.count() ) ]
+        
+    
+    def _UpdateResults( self ):
+        
+        texts = self._GetStartingTexts()
+        
+        try:
+            
+            results = self._string_processor.ProcessStrings( texts )
+            
+        except HydrusExceptions.ParseException as e:
+            
+            results = [ 'error in processing: {}'.format( e ) ]
+            
+        
+        self._result_data.clear()
+        
+        for ( insertion_index, result ) in enumerate( results ):
+            
+            item = QW.QListWidgetItem()
+            
+            item.setText( result )
+            item.setData( QC.Qt.UserRole, result )
+            
+            self._result_data.insertItem( insertion_index, item )
+            
+        
+    
+    def EventSelection( self ):
+        
+        items = self._test_data.selectedItems()
+        
+        if len( items ) == 1:
+            
+            ( list_widget_item, ) = items
+            
+            text = list_widget_item.data( QC.Qt.UserRole )
+            
+            self.textSelected.emit( text )
+            
+        
+    
+    def GetResultTexts( self, step_index ):
+        
+        texts = self._GetStartingTexts()
+        
+        try:
+            
+            results = self._string_processor.ProcessStrings( texts, max_steps_allowed = step_index + 1 )
+            
+        except:
+            
+            results = []
+            
+        
+        return results
+        
+    
+    def SetStringProcessor( self, string_processor: ClientParsing.StringProcessor ):
+        
+        self._string_processor = string_processor
+        
+        self._UpdateResults()
+        
+    
+    def SetTestData( self, test_data: ClientParsing.ParsingTestData ):
+        
+        self._test_data.clear()
+        
+        for ( insertion_index, text ) in enumerate( test_data.texts ):
+            
+            item = QW.QListWidgetItem()
+            
+            item.setText( text )
+            item.setData( QC.Qt.UserRole, text )
+            
+            self._test_data.insertItem( insertion_index, item )
+            
+        
+        self._UpdateResults()
+        
+        if len( test_data.texts ) > 0:
+            
+            self._test_data.item( 0 ).setSelected( False )
+            self._test_data.item( 0 ).setSelected( True )
+            
+            #self.textSelected.emit( self._test_data.item( 0 ).data( QC.Qt.UserRole ) )
+            
+        
+    
+class SingleStringConversionTestPanel( QW.QWidget ):
+    
+    def __init__( self, parent: QW.QWidget, string_processor: ClientParsing.StringProcessor ):
+        
+        QW.QWidget.__init__( self, parent )
+        
+        self._string_processor = string_processor
+        
+        self._example_string = QW.QLineEdit( self )
+        
+        self._example_results = ClientGUICommon.BetterNotebook( self )
+        
+        #
+        
+        vbox = QP.VBoxLayout()
+        
+        QP.AddToLayout( vbox, ClientGUICommon.BetterStaticText( self, label = 'single example string' ), CC.FLAGS_CENTER )
+        QP.AddToLayout( vbox, self._example_string, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
+        QP.AddToLayout( vbox, ClientGUICommon.BetterStaticText( self, label = 'results for each step' ), CC.FLAGS_CENTER )
+        QP.AddToLayout( vbox, self._example_results, CC.FLAGS_EXPAND_BOTH_WAYS )
+        
+        self.setLayout( vbox )
+        
+        self._example_string.textChanged.connect( self._UpdateResults )
+        
+    
+    def _UpdateResults( self ):
+        
+        processing_steps = self._string_processor.GetProcessingSteps()
+        
+        current_selected_index = self._example_results.currentIndex()
+        
+        self._example_results.DeleteAllPages()
+        
+        example_string = self._example_string.text()
+        
+        stop_now = False
+        
+        for i in range( len( processing_steps ) ):
+            
+            if isinstance( processing_steps[i], ClientParsing.StringSlicer ):
+                
+                continue
+                
+            
+            try:
+                
+                results = self._string_processor.ProcessStrings( [ example_string ], max_steps_allowed = i + 1, no_slicing = True )
+                
+            except Exception as e:
+                
+                results = [ 'error: {}'.format( str( e ) ) ]
+                
+                stop_now = True
+                
+            
+            results_list = QW.QListWidget( self._example_results )
+            results_list.setSelectionMode( QW.QListWidget.NoSelection )  
+            
+            if len( results ) == 0:
+                
+                results_list.addItem( NO_RESULTS_TEXT )
+                
+                stop_now = True
+                
+            else:
+                
+                for result in results:
+                    
+                    if not isinstance( result, str ):
+                        
+                        result = repr( result )
+                        
+                    
+                    results_list.addItem( result )
+                    
+                
+            
+            tab_label = '{} ({})'.format( processing_steps[i].ToString( simple = True ), HydrusData.ToHumanInt( len( results ) ) )
+            
+            self._example_results.addTab( results_list, tab_label )
+            
+            if stop_now:
+                
+                break
+                
+            
+        
+        if self._example_results.count() > current_selected_index:
+            
+            self._example_results.setCurrentIndex( current_selected_index )
+            
+        
+    
+    def GetResultText( self, step_index: int ):
+        
+        example_text = self._example_string.text()
+        
+        if 0 < step_index < self._example_results.count() + 1:
+            
+            try:
+                
+                t = self._example_results.widget( step_index - 1 ).item( 0 ).text()
+                
+                if t != NO_RESULTS_TEXT:
+                    
+                    example_text = t
+                    
+                
+            except:
+                
+                pass
+                
+            
+        
+        return example_text
+        
+    
+    def GetStartingText( self ):
+        
+        return self._example_string.text()
+        
+    
+    def SetStringProcessor( self, string_processor: ClientParsing.StringProcessor ):
+        
+        self._string_processor = string_processor
+        
+        if True in ( isinstance( processing_step, ClientParsing.StringSlicer ) for processing_step in self._string_processor.GetProcessingSteps() ):
+            
+            self.setToolTip( 'String Slicing is ignored here.' )
+            
+        else:
+            
+            self.setToolTip( '' )
+            
+        
+        self._UpdateResults()
+        
+    
+    def SetExampleString( self, example_string: str ):
+        
+        self._example_string.setText( example_string )
+        
+        self._UpdateResults()
+        
+    
 class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
     
     def __init__( self, parent: QW.QWidget, string_converter: ClientParsing.StringConverter, example_string_override = None ):
@@ -1011,6 +1291,199 @@ class EditStringMatchPanel( ClientGUIScrolledPanels.EditPanel ):
         self._UpdateControlVisibility()
         
     
+SELECT_SINGLE = 0
+SELECT_RANGE = 1
+
+class EditStringSlicerPanel( ClientGUIScrolledPanels.EditPanel ):
+    
+    def __init__( self, parent, string_slicer: ClientParsing.StringSlicer, test_data: typing.Sequence[ str ] = [] ):
+        
+        ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
+        
+        #
+        
+        self._controls_panel = ClientGUICommon.StaticBox( self, 'selector values' )
+        
+        self._select_type = ClientGUICommon.BetterChoice( self._controls_panel )
+        
+        self._select_type.addItem( 'select one item', SELECT_SINGLE )
+        self._select_type.addItem( 'select range', SELECT_RANGE )
+        
+        self._single_panel = QW.QWidget( self._controls_panel )
+        
+        self._index_single = QP.MakeQSpinBox( self._single_panel, min = -65536, max = 65536 )
+        
+        self._range_panel = QW.QWidget( self._controls_panel )
+        
+        self._index_start = ClientGUICommon.NoneableSpinCtrl( self._range_panel, none_phrase = 'start at the beginning', min = -65536, max = 65536)
+        self._index_end = ClientGUICommon.NoneableSpinCtrl( self._range_panel, none_phrase = 'finish at the end', min = -65536, max = 65536)
+        
+        self._summary_st = ClientGUICommon.BetterStaticText( self._controls_panel )
+        
+        #
+        
+        self._example_panel = ClientGUICommon.StaticBox( self, 'test results' )
+        
+        self._example_strings = QW.QListWidget( self._example_panel )
+        self._example_strings.setSelectionMode( QW.QListWidget.NoSelection )
+        
+        self._example_strings_sliced = QW.QListWidget( self._example_panel )
+        self._example_strings_sliced.setSelectionMode( QW.QListWidget.NoSelection )
+        
+        #
+        
+        for s in test_data:
+            
+            self._example_strings.addItem( s )
+            
+        
+        self.SetValue( string_slicer )
+        
+        #
+        
+        rows = []
+        
+        rows.append( ( 'index to select: ', self._index_single ) )
+        
+        gridbox = ClientGUICommon.WrapInGrid( self._single_panel, rows )
+        
+        self._single_panel.setLayout( gridbox )
+        
+        rows = []
+        
+        rows.append( ( 'starting index: ', self._index_start ) )
+        rows.append( ( 'ending index: ', self._index_end ) )
+        
+        gridbox = ClientGUICommon.WrapInGrid( self._range_panel, rows )
+        
+        self._range_panel.setLayout( gridbox )
+        
+        st = ClientGUICommon.BetterStaticText( self._controls_panel, label = 'Negative indices are ok! Check the summary text to make sure your numbers are correct!' )
+        
+        st.setWordWrap( True )
+        
+        self._controls_panel.Add( st, CC.FLAGS_EXPAND_PERPENDICULAR )
+        self._controls_panel.Add( self._select_type, CC.FLAGS_EXPAND_PERPENDICULAR )
+        self._controls_panel.Add( self._single_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
+        self._controls_panel.Add( self._range_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
+        self._controls_panel.Add( self._summary_st, CC.FLAGS_CENTER )
+        
+        hbox = QP.HBoxLayout()
+        
+        QP.AddToLayout( hbox, self._example_strings, CC.FLAGS_EXPAND_BOTH_WAYS )
+        QP.AddToLayout( hbox, self._example_strings_sliced, CC.FLAGS_EXPAND_BOTH_WAYS )
+        
+        self._example_panel.Add( hbox, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
+        
+        vbox = QP.VBoxLayout()
+        
+        QP.AddToLayout( vbox, self._controls_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
+        QP.AddToLayout( vbox, self._example_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
+        
+        self.widget().setLayout( vbox )
+        
+        #
+        
+        self.SetValue( string_slicer )
+        
+        self._select_type.currentIndexChanged.connect( self._ShowHideControls )
+        
+        self._index_single.valueChanged.connect( self._UpdateControls )
+        self._index_start.valueChanged.connect( self._UpdateControls )
+        self._index_end.valueChanged.connect( self._UpdateControls )
+        
+    
+    def _GetValue( self ):
+        
+        select_type = self._select_type.GetValue()
+        
+        if select_type == SELECT_SINGLE:
+            
+            index_start = self._index_single.value()
+            
+            if index_start == -1:
+                
+                index_end = None
+                
+            else:
+                
+                index_end = index_start + 1
+                
+            
+        elif select_type == SELECT_RANGE:
+            
+            index_start = self._index_start.GetValue()
+            index_end = self._index_end.GetValue()
+            
+        
+        string_slicer = ClientParsing.StringSlicer( index_start = index_start, index_end = index_end )
+        
+        return string_slicer
+        
+    
+    def _ShowHideControls( self ):
+        
+        select_type = self._select_type.GetValue()
+        
+        self._single_panel.setVisible( select_type == SELECT_SINGLE )
+        self._range_panel.setVisible( select_type == SELECT_RANGE )
+        
+        self._UpdateControls()
+        
+    
+    def _UpdateControls( self ):
+        
+        string_slicer = self._GetValue()
+        
+        self._summary_st.setText( string_slicer.ToString() )
+        
+        texts = [ self._example_strings.item( i ).text() for i in range( self._example_strings.count() ) ]
+        
+        try:
+            
+            sliced_texts = string_slicer.Slice( texts )
+            
+        except Exception as e:
+            
+            sliced_texts = [ 'Error: {}'.format( e ) ]
+            
+        
+        self._example_strings_sliced.clear()
+        
+        for s in sliced_texts:
+            
+            self._example_strings_sliced.addItem( s )
+            
+        
+    
+    def GetValue( self ):
+        
+        string_slicer = self._GetValue()
+        
+        return string_slicer
+        
+    
+    def SetValue( self, string_slicer: ClientParsing.StringSlicer ):
+        
+        ( index_start, index_end ) = string_slicer.GetIndexStartEnd()
+        
+        self._index_single.setValue( index_start if index_start is not None else 0 )
+        
+        self._index_start.SetValue( index_start )
+        self._index_end.SetValue( index_end )
+        
+        if string_slicer.SelectsOne():
+            
+            self._select_type.SetValue( SELECT_SINGLE )
+            
+        else:
+            
+            self._select_type.SetValue( SELECT_RANGE )
+            
+        
+        self._ShowHideControls()
+        
+    
 class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
     
     def __init__( self, parent, string_sorter: ClientParsing.StringSorter, test_data: typing.Sequence[ str ] = [] ):
@@ -1019,13 +1492,15 @@ class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
         
         #
         
-        self._controls_panel = ClientGUICommon.StaticBox( self, 'splitter values' )
+        self._controls_panel = ClientGUICommon.StaticBox( self, 'sort values' )
         
         self._sort_type = ClientGUICommon.BetterChoice( self._controls_panel )
         
         self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT ], ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT )
         self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC ], ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC )
         
+        tt = 'Human sort sorts numbers as you understand them. "image 2" comes before "image 10". Lexicographic compares each character in turn. "image 02" comes before "image 10", which comes before "image 2".'
+        
         self._asc = QW.QCheckBox( self._controls_panel )
         
         self._regex = ClientGUICommon.NoneableTextCtrl( self._controls_panel, none_phrase = 'use whole string' )
@@ -1074,13 +1549,6 @@ class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
         
         vbox = QP.VBoxLayout()
         
-        woah = 'Because string processing UI is still built around a single \'example\' string, the test UI here is bad! The whole system will be eventually updated to handle newlines and multiple example strings.'
-        
-        st = ClientGUICommon.BetterStaticText( self, label = woah )
-        
-        st.setWordWrap( True )
-        
-        QP.AddToLayout( vbox, st, CC.FLAGS_EXPAND_PERPENDICULAR )
         QP.AddToLayout( vbox, self._controls_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
         QP.AddToLayout( vbox, self._example_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
         
@@ -1152,9 +1620,9 @@ class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
     
     def GetValue( self ):
         
-        string_match = self._GetValue()
+        string_sorter = self._GetValue()
         
-        return string_match
+        return string_sorter
         
     
     def SetValue( self, string_sorter: ClientParsing.StringSorter ):
@@ -1259,9 +1727,9 @@ class EditStringSplitterPanel( ClientGUIScrolledPanels.EditPanel ):
     
     def GetValue( self ):
         
-        string_match = self._GetValue()
+        string_splitter = self._GetValue()
         
-        return string_match
+        return string_splitter
         
     
     def SetValue( self, string_splitter: ClientParsing.StringSplitter ):
@@ -1277,8 +1745,6 @@ class EditStringSplitterPanel( ClientGUIScrolledPanels.EditPanel ):
     
 class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
     
-    NO_RESULTS_TEXT = 'no results'
-    
     def __init__( self, parent, string_processor: ClientParsing.StringProcessor, test_data: ClientParsing.ParsingTestData ):
         
         ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
@@ -1293,9 +1759,11 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
         
         self._example_panel = ClientGUICommon.StaticBox( self, 'test results' )
         
-        self._example_string = QW.QLineEdit( self._example_panel )
+        self._multiline_test_panel = MultilineStringConversionTestPanel( self._example_panel, string_processor )
         
-        self._example_results = ClientGUICommon.BetterNotebook( self._example_panel )
+        self._single_test_panel = SingleStringConversionTestPanel( self._example_panel, string_processor )
+        
+        #
         
         ( w, h ) = ClientGUIFunctions.ConvertTextToPixels( self._example_panel, ( 64, 24 ) )
         
@@ -1303,41 +1771,31 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
         
         #
         
-        if len( test_data.texts ) > 0:
-            
-            example_string = test_data.texts[0]
-            
-        else:
-            example_string = ''
-            
-        
-        self._example_string.setText( example_string )
+        self._controls_panel.Add( self._processing_steps, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
         
         #
         
-        self._controls_panel.Add( self._processing_steps, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
+        example_hbox = QP.HBoxLayout()
         
-        rows = []
+        QP.AddToLayout( example_hbox, self._multiline_test_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
+        QP.AddToLayout( example_hbox, self._single_test_panel, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
         
-        rows.append( ( 'example string: ', self._example_string ) )
+        self._example_panel.Add( example_hbox, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
         
-        gridbox = ClientGUICommon.WrapInGrid( self._example_panel, rows )
+        vbox = QP.VBoxLayout()
         
-        self._example_panel.Add( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
-        self._example_panel.Add( ClientGUICommon.BetterStaticText( self, label = 'result:' ), CC.FLAGS_EXPAND_PERPENDICULAR )
-        self._example_panel.Add( self._example_results, CC.FLAGS_EXPAND_BOTH_WAYS )
+        QP.AddToLayout( vbox, self._controls_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
+        QP.AddToLayout( vbox, self._example_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
         
-        hbox = QP.VBoxLayout()
-        
-        QP.AddToLayout( hbox, self._controls_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
-        QP.AddToLayout( hbox, self._example_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
-        
-        self.widget().setLayout( hbox )
+        self.widget().setLayout( vbox )
         
         #
         
         self._processing_steps.listBoxChanged.connect( self._UpdateControls )
-        self._example_string.textChanged.connect( self._UpdateControls )
+        
+        self._multiline_test_panel.textSelected.connect( self._single_test_panel.SetExampleString )
+        
+        self._multiline_test_panel.SetTestData( test_data )
         
         self.SetValue( string_processor )
         
@@ -1348,7 +1806,8 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
             ( 'String Match', ClientParsing.StringMatch, 'An object that filters strings.' ),
             ( 'String Converter', ClientParsing.StringConverter, 'An object that converts strings from one thing to another.' ),
             ( 'String Splitter', ClientParsing.StringSplitter, 'An object that breaks strings into smaller strings.' ),
-            ( 'String Sorter', ClientParsing.StringSorter, 'An object that sorts strings.' )
+            ( 'String Sorter', ClientParsing.StringSorter, 'An object that sorts strings.' ),
+            ( 'String Selector/Slicer', ClientParsing.StringSlicer, 'An object that filter-selects from the list of strings. Either absolute index position or a range.' )
         ]
         
         try:
@@ -1362,7 +1821,9 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
         
         if string_processing_step_type == ClientParsing.StringMatch:
             
-            string_processing_step = ClientParsing.StringMatch( example_string = self._example_string.text() )
+            example_text = self._single_test_panel.GetStartingText()
+            
+            string_processing_step = ClientParsing.StringMatch( example_string = example_text )
             
             example_text = self._GetExampleTextForStringProcessingStep( string_processing_step )
             
@@ -1402,6 +1863,12 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
                 
                 panel = EditStringSorterPanel( dlg, string_processing_step, test_data = test_data )
                 
+            elif isinstance( string_processing_step, ClientParsing.StringSlicer ):
+                
+                test_data = self._GetExampleTextsForStringSorter( string_processing_step )
+                
+                panel = EditStringSlicerPanel( dlg, string_processing_step, test_data = test_data )
+                
             
             dlg.SetPanel( panel )
             
@@ -1425,7 +1892,7 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
     
     def _GetExampleTextForStringProcessingStep( self, string_processing_step: ClientParsing.StringProcessingStep ):
         
-        # ultimately rework this to multiline test_data m8
+        # ultimately rework this to multiline test_data m8, but the panels need it first
         
         current_string_processor = self._GetValue()
         
@@ -1440,24 +1907,7 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
             example_text_index = len( current_string_processing_steps )
             
         
-        example_text = self._example_string.text()
-        
-        if 0 < example_text_index < self._example_results.count() + 1:
-            
-            try:
-                
-                t = self._example_results.widget( example_text_index - 1 ).item( 0 ).text()
-                
-                if t != self.NO_RESULTS_TEXT:
-                    
-                    example_text = t
-                    
-                
-            except:
-                
-                pass
-                
-            
+        example_text = self._single_test_panel.GetResultText( example_text_index )
         
         return example_text
         
@@ -1479,26 +1929,7 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
             example_text_index = len( current_string_processing_steps )
             
         
-        example_texts = [ self._example_string.text() ]
-        
-        if 0 < example_text_index < self._example_results.count() + 1:
-            
-            try:
-                
-                widget = self._example_results.widget( example_text_index - 1 )
-                
-                example_texts = [ widget.item( i ).text() for i in range( widget.count() ) ]
-                
-                if example_texts == [ self.NO_RESULTS_TEXT ]:
-                    
-                    example_texts = [ self._example_string.text() ]
-                    
-                
-            except:
-                
-                pass
-                
-            
+        example_texts = self._multiline_test_panel.GetResultTexts( example_text_index )
         
         return example_texts
         
@@ -1518,65 +1949,8 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
         
         string_processor = self._GetValue()
         
-        processing_steps = string_processor.GetProcessingSteps()
-        
-        current_selected_index = self._example_results.currentIndex()
-        
-        self._example_results.DeleteAllPages()
-        
-        example_string = self._example_string.text()
-        
-        stop_now = False
-        
-        for i in range( len( processing_steps ) ):
-            
-            try:
-                
-                results = string_processor.ProcessStrings( [ example_string ], max_steps_allowed = i + 1 )
-                
-            except Exception as e:
-                
-                results = [ 'error: {}'.format( str( e ) ) ]
-                
-                stop_now = True
-                
-            
-            results_list = QW.QListWidget( self._example_panel )
-            results_list.setSelectionMode( QW.QListWidget.NoSelection )  
-            
-            if len( results ) == 0:
-                
-                results_list.addItem( self.NO_RESULTS_TEXT )
-                
-                stop_now = True
-                
-            else:
-                
-                for result in results:
-                    
-                    if not isinstance( result, str ):
-                        
-                        result = repr( result )
-                        
-                    
-                    results_list.addItem( result )
-                    
-                
-            
-            tab_label = '{} ({})'.format( processing_steps[i].ToString( simple = True ), HydrusData.ToHumanInt( len( results ) ) )
-            
-            self._example_results.addTab( results_list, tab_label )
-            
-            if stop_now:
-                
-                break
-                
-            
-        
-        if self._example_results.count() > current_selected_index:
-            
-            self._example_results.setCurrentIndex( current_selected_index )
-            
+        self._multiline_test_panel.SetStringProcessor( string_processor )
+        self._single_test_panel.SetStringProcessor( string_processor )
         
     
     def GetValue( self ):

hydrus/client/gui/ClientGUITagSorting.py

@@ -0,0 +1,146 @@
+import typing
+
+from qtpy import QtCore as QC
+from qtpy import QtWidgets as QW
+
+from hydrus.client import ClientConstants as CC
+from hydrus.client.gui import ClientGUICommon
+from hydrus.client.gui import QtPorting as QP
+from hydrus.client.metadata import ClientTagSorting
+
+class TagSortControl( QW.QWidget ):
+    
+    valueChanged = QC.Signal()
+    
+    def __init__( self, parent: QW.QWidget, tag_sort: ClientTagSorting.TagSort, show_siblings = False ):
+        
+        QW.QWidget.__init__( self, parent )
+        
+        self._sort_type = ClientGUICommon.BetterChoice( self )
+        
+        for sort_type in ( ClientTagSorting.SORT_BY_HUMAN_TAG, ClientTagSorting.SORT_BY_HUMAN_SUBTAG, ClientTagSorting.SORT_BY_COUNT ):
+            
+            self._sort_type.addItem( ClientTagSorting.sort_type_str_lookup[ sort_type ], sort_type )
+            
+        
+        self._sort_order_text = ClientGUICommon.BetterChoice( self )
+        
+        self._sort_order_text.addItem( 'a-z', CC.SORT_ASC )
+        self._sort_order_text.addItem( 'z-a', CC.SORT_DESC )
+        
+        self._sort_order_count = ClientGUICommon.BetterChoice( self )
+        
+        self._sort_order_count.addItem( 'most first', CC.SORT_DESC )
+        self._sort_order_count.addItem( 'fewest first', CC.SORT_ASC )
+        
+        self._show_siblings = show_siblings
+        
+        self._use_siblings = ClientGUICommon.BetterChoice( self )
+        
+        self._use_siblings.addItem( 'siblings', True )
+        self._use_siblings.addItem( 'tags', False )
+        
+        self._group_by = ClientGUICommon.BetterChoice( self )
+        
+        self._group_by.addItem( 'no grouping', ClientTagSorting.GROUP_BY_NOTHING )
+        self._group_by.addItem( 'group namespace', ClientTagSorting.GROUP_BY_NAMESPACE )
+        
+        #
+        
+        self.SetValue( tag_sort )
+        
+        #
+        
+        hbox = QP.HBoxLayout()
+        
+        QP.AddToLayout( hbox, self._sort_type, CC.FLAGS_EXPAND_BOTH_WAYS )
+        QP.AddToLayout( hbox, self._sort_order_text, CC.FLAGS_CENTER_PERPENDICULAR )
+        QP.AddToLayout( hbox, self._sort_order_count, CC.FLAGS_CENTER_PERPENDICULAR )
+        QP.AddToLayout( hbox, self._use_siblings, CC.FLAGS_CENTER_PERPENDICULAR )
+        QP.AddToLayout( hbox, self._group_by, CC.FLAGS_CENTER_PERPENDICULAR )
+        
+        self.setLayout( hbox )
+        
+        #
+        
+        self._sort_type.currentIndexChanged.connect( self._UpdateControlsAfterSortTypeChanged )
+        
+        self._sort_type.currentIndexChanged.connect( self._HandleValueChanged )
+        self._sort_order_text.currentIndexChanged.connect( self._HandleValueChanged )
+        self._sort_order_count.currentIndexChanged.connect( self._HandleValueChanged )
+        self._use_siblings.currentIndexChanged.connect( self._HandleValueChanged )
+        self._group_by.currentIndexChanged.connect( self._HandleValueChanged )
+        
+    
+    def _UpdateControlsAfterSortTypeChanged( self ):
+        
+        sort_type = self._sort_type.GetValue()
+        
+        self._sort_order_text.setVisible( sort_type in ( ClientTagSorting.SORT_BY_HUMAN_TAG, ClientTagSorting.SORT_BY_HUMAN_SUBTAG ) )
+        self._sort_order_count.setVisible( sort_type == ClientTagSorting.SORT_BY_COUNT )
+        
+        self._use_siblings.setVisible( self._show_siblings and sort_type != ClientTagSorting.SORT_BY_COUNT )
+        self._group_by.setVisible( sort_type != ClientTagSorting.SORT_BY_HUMAN_SUBTAG )
+        
+    
+    def _HandleValueChanged( self ):
+        
+        self.valueChanged.emit()
+        
+    
+    def GetValue( self ):
+        
+        sort_type = self._sort_type.GetValue()
+        
+        if sort_type == ClientTagSorting.SORT_BY_COUNT:
+            
+            sort_order = self._sort_order_count.GetValue()
+            
+        else:
+            
+            sort_order = self._sort_order_text.GetValue()
+            
+        
+        tag_sort = ClientTagSorting.TagSort(
+            sort_type = sort_type,
+            sort_order = sort_order,
+            use_siblings = self._use_siblings.GetValue(),
+            group_by = self._group_by.GetValue()
+        )
+        
+        return tag_sort
+        
+    
+    def SetValue( self, tag_sort: ClientTagSorting.TagSort ):
+        
+        self._sort_type.blockSignals( True )
+        self._sort_order_text.blockSignals( True )
+        self._sort_order_count.blockSignals( True )
+        self._use_siblings.blockSignals( True )
+        self._group_by.blockSignals( True )
+        
+        self._sort_type.SetValue( tag_sort.sort_type )
+        
+        if tag_sort.sort_type == ClientTagSorting.SORT_BY_COUNT:
+            
+            self._sort_order_count.SetValue( tag_sort.sort_order )
+            
+        else:
+            
+            self._sort_order_text.SetValue( tag_sort.sort_order )
+            
+        
+        self._use_siblings.SetValue( tag_sort.use_siblings )
+        self._group_by.SetValue( tag_sort.group_by )
+        
+        self._sort_type.blockSignals( False )
+        self._sort_order_text.blockSignals( False )
+        self._sort_order_count.blockSignals( False )
+        self._use_siblings.blockSignals( False )
+        self._group_by.blockSignals( False )
+        
+        self._UpdateControlsAfterSortTypeChanged()
+        
+        self._HandleValueChanged()
+        
+    

hydrus/client/gui/ClientGUITagSuggestions.py

@@ -184,7 +184,7 @@ class FavouritesTagsPanel( QW.QWidget ):
         
         favourites = list( HG.client_controller.new_options.GetSuggestedTagsFavourites( self._service_key ) )
         
-        ClientTagSorting.SortTags( HC.options[ 'default_tag_sort' ], favourites )
+        ClientTagSorting.SortTags( HG.client_controller.new_options.GetDefaultTagSort(), favourites )
         
         tags = FilterSuggestedTagsForMedia( favourites, self._media, self._service_key )
         

hydrus/client/gui/ClientGUITags.py

@@ -2009,7 +2009,7 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
             
             self._i_am_local_tag_service = self._service.GetServiceType() == HC.LOCAL_TAG
             
-            self._tags_box_sorter = ClientGUIListBoxes.StaticBoxSorterForListBoxTags( self, 'tags' )
+            self._tags_box_sorter = ClientGUIListBoxes.StaticBoxSorterForListBoxTags( self, 'tags', show_siblings_sort = True )
             
             self._tags_box = ClientGUIListBoxes.ListBoxTagsMediaTagsDialog( self._tags_box_sorter, self.EnterTags, self.RemoveTags )
             

hydrus/client/gui/lists/ClientGUIListBoxes.py

@@ -24,6 +24,7 @@ from hydrus.client.gui import ClientGUICore as CGC
 from hydrus.client.gui import ClientGUIFunctions
 from hydrus.client.gui import ClientGUIMenus
 from hydrus.client.gui import ClientGUIShortcuts
+from hydrus.client.gui import ClientGUITagSorting
 from hydrus.client.gui import QtPorting as QP
 from hydrus.client.gui.search import ClientGUISearch
 from hydrus.client.gui.lists import ClientGUIListBoxesData
@@ -2554,9 +2555,11 @@ class ListBoxTags( ListBox ):
                                 service_key_groups_to_tags[ tuple( s_ks ) ].append( t )
                                 
                             
+                            tag_sort = ClientTagSorting.TagSort.STATICGetTextASCDefault()
+                            
                             for t_list in service_key_groups_to_tags.values():
                                 
-                                ClientTagSorting.SortTags( CC.SORT_BY_LEXICOGRAPHIC_ASC, t_list )
+                                ClientTagSorting.SortTags( tag_sort, t_list )
                                 
                             
                             service_key_groups = sorted( service_key_groups_to_tags.keys(), key = lambda s_k_g: ( -len( s_k_g ), convert_service_keys_to_name_string( s_k_g ) ) )
@@ -2582,10 +2585,12 @@ class ListBoxTags( ListBox ):
                                 service_key_groups_to_tags[ tuple( s_ks ) ][1].append( c )
                                 
                             
+                            tag_sort = ClientTagSorting.TagSort.STATICGetTextASCDefault()
+                            
                             for ( t_list_1, t_list_2 ) in service_key_groups_to_tags.values():
                                 
-                                ClientTagSorting.SortTags( CC.SORT_BY_LEXICOGRAPHIC_ASC, t_list_1 )
-                                ClientTagSorting.SortTags( CC.SORT_BY_LEXICOGRAPHIC_ASC, t_list_2 )
+                                ClientTagSorting.SortTags( tag_sort, t_list_1 )
+                                ClientTagSorting.SortTags( tag_sort, t_list_2 )
                                 
                             
                             service_key_groups = sorted( service_key_groups_to_tags.keys(), key = lambda s_k_g: ( -len( s_k_g ), convert_service_keys_to_name_string( s_k_g ) ) )
@@ -3420,7 +3425,7 @@ class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
         
         ListBoxTagsDisplayCapable.__init__( self, parent, service_key = service_key, tag_display_type = tag_display_type, height_num_chars = 24 )
         
-        self._sort = HC.options[ 'default_tag_sort' ]
+        self._tag_sort = HG.client_controller.new_options.GetDefaultTagSort()
         
         self._last_media = set()
         
@@ -3521,7 +3526,7 @@ class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
             ( self._show_petitioned, self._petitioned_tags_to_count )
         ]
         
-        counts_to_include = [ c for ( show, c ) in jobs ]
+        counts_to_include = [ c for ( show, c ) in jobs if show ]
         
         for term in self._ordered_terms:
             
@@ -3532,16 +3537,15 @@ class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
             terms_to_count[ term ] = count
             
         
+        item_to_tag_key_wrapper = lambda term: term.GetTag()
+        item_to_sibling_key_wrapper = item_to_tag_key_wrapper
+        
         if self._sibling_decoration_allowed:
             
-            item_to_tag_key_wrapper = lambda term: term.GetBestTag()
-            
-        else:
-            
-            item_to_tag_key_wrapper = lambda term: term.GetTag()
+            item_to_sibling_key_wrapper = lambda term: term.GetBestTag()
             
         
-        ClientTagSorting.SortTags( self._sort, self._ordered_terms, tag_items_to_count = terms_to_count, item_to_tag_key_wrapper = item_to_tag_key_wrapper )
+        ClientTagSorting.SortTags( self._tag_sort, self._ordered_terms, tag_items_to_count = terms_to_count, item_to_tag_key_wrapper = item_to_tag_key_wrapper, item_to_sibling_key_wrapper = item_to_sibling_key_wrapper )
         
         self._RegenTermsToIndices()
         
@@ -3553,9 +3557,9 @@ class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
         self.SetTagsByMedia( self._last_media )
         
     
-    def SetSort( self, sort ):
+    def SetSort( self, tag_sort: ClientTagSorting.TagSort ):
         
-        self._sort = sort
+        self._tag_sort = tag_sort
         
         self._Sort()
         
@@ -3686,28 +3690,16 @@ class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
     
 class StaticBoxSorterForListBoxTags( ClientGUICommon.StaticBox ):
     
-    def __init__( self, parent, title ):
+    def __init__( self, parent, title, show_siblings_sort = False ):
         
         ClientGUICommon.StaticBox.__init__( self, parent, title )
         
-        self._sorter = ClientGUICommon.BetterChoice( self )
+        # make this its own panel
+        self._tag_sort = ClientGUITagSorting.TagSortControl( self, HG.client_controller.new_options.GetDefaultTagSort(), show_siblings = show_siblings_sort )
         
-        self._sorter.addItem( 'lexicographic (a-z)', CC.SORT_BY_LEXICOGRAPHIC_ASC )
-        self._sorter.addItem( 'lexicographic (z-a)', CC.SORT_BY_LEXICOGRAPHIC_DESC )
-        self._sorter.addItem( 'lexicographic (a-z) (group unnamespaced)', CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC )
-        self._sorter.addItem( 'lexicographic (z-a) (group unnamespaced)', CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC )
-        self._sorter.addItem( 'lexicographic (a-z) (ignore namespace)', CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC )
-        self._sorter.addItem( 'lexicographic (z-a) (ignore namespace)', CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC )
-        self._sorter.addItem( 'incidence (desc)', CC.SORT_BY_INCIDENCE_DESC )
-        self._sorter.addItem( 'incidence (asc)', CC.SORT_BY_INCIDENCE_ASC )
-        self._sorter.addItem( 'incidence (desc) (grouped by namespace)', CC.SORT_BY_INCIDENCE_NAMESPACE_DESC )
-        self._sorter.addItem( 'incidence (asc) (grouped by namespace)', CC.SORT_BY_INCIDENCE_NAMESPACE_ASC )
+        self._tag_sort.valueChanged.connect( self.EventSort )
         
-        self._sorter.SetValue( HC.options[ 'default_tag_sort' ] )
-        
-        self._sorter.currentIndexChanged.connect( self.EventSort )
-        
-        self.Add( self._sorter, CC.FLAGS_EXPAND_PERPENDICULAR )
+        self.Add( self._tag_sort, CC.FLAGS_EXPAND_PERPENDICULAR )
         
     
     def SetTagServiceKey( self, service_key ):
@@ -3715,16 +3707,11 @@ class StaticBoxSorterForListBoxTags( ClientGUICommon.StaticBox ):
         self._tags_box.SetTagServiceKey( service_key )
         
     
-    def EventSort( self, index ):
+    def EventSort( self ):
         
-        selection = self._sorter.currentIndex()
+        sort = self._tag_sort.GetValue()
         
-        if selection != -1:
-            
-            sort = self._sorter.GetValue()
-            
-            self._tags_box.SetSort( sort )
-            
+        self._tags_box.SetSort( sort )
         
     
     def SetTagsBox( self, tags_box: ListBoxTagsMedia ):

hydrus/client/importing/ClientImportFileSeeds.py

@@ -90,23 +90,38 @@ class FileImportJob( object ):
             
             self.GenerateInfo()
             
-            self.CheckIsGoodToImport()
-            
-            mime = self.GetMime()
-            
-            if status_hook is not None:
+            try:
                 
-                status_hook( 'copying file' )
+                self.CheckIsGoodToImport()
+                
+                ok_to_go = True
+                
+            except HydrusExceptions.FileSizeException as e:
+                
+                ok_to_go = False
+                
+                import_status = CC.STATUS_SKIPPED
+                note = str( e )
                 
             
-            HG.client_controller.client_files_manager.AddFile( hash, mime, self._temp_path, thumbnail_bytes = self._thumbnail_bytes )
-            
-            if status_hook is not None:
+            if ok_to_go:
                 
-                status_hook( 'updating database' )
+                mime = self.GetMime()
+                
+                if status_hook is not None:
+                    
+                    status_hook( 'copying file' )
+                    
+                
+                HG.client_controller.client_files_manager.AddFile( hash, mime, self._temp_path, thumbnail_bytes = self._thumbnail_bytes )
+                
+                if status_hook is not None:
+                    
+                    status_hook( 'updating database' )
+                    
+                
+                ( import_status, note ) = HG.client_controller.WriteSynchronous( 'import_file', self )
                 
-            
-            ( import_status, note ) = HG.client_controller.WriteSynchronous( 'import_file', self )
             
         else:
             

hydrus/client/importing/ClientImportSubscriptions.py

@@ -659,6 +659,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
         gallery_seed_log = query_log_container.GetGallerySeedLog()
         
         this_is_initial_sync = query_header.IsInitialSync()
+        num_file_seeds_at_start = len( file_seed_cache )
         total_new_urls_for_this_sync = 0
         total_already_in_urls_for_this_sync = 0
         
@@ -803,37 +804,55 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
                             file_seeds_to_add_ordered.append( file_seed )
                             
                         
-                        if file_limit_for_this_sync is not None and total_new_urls_for_this_sync + num_urls_added >= file_limit_for_this_sync:
+                        if file_limit_for_this_sync is not None:
                             
-                            # we have found enough new files this sync, so should stop adding files and new gallery pages
-                            
-                            if this_is_initial_sync:
+                            if total_new_urls_for_this_sync + num_urls_added >= file_limit_for_this_sync:
                                 
-                                stop_reason = 'hit initial file limit'
+                                # we have found enough new files this sync, so should stop adding files and new gallery pages
                                 
-                            else:
-                                
-                                if total_already_in_urls_for_this_sync + num_urls_already_in_file_seed_cache > 0:
+                                if this_is_initial_sync:
                                     
-                                    # this sync produced some knowns, so it is likely we have stepped through a mix of old and tagged-late new files
-                                    # we might also be on the second sync with a periodic limit greater than the initial limit
-                                    # either way, this is no reason to go crying to the user
-                                    
-                                    stop_reason = 'hit periodic file limit after seeing several already-seen files'
+                                    stop_reason = 'hit initial file limit'
                                     
                                 else:
                                     
-                                    # this page had all entirely new files
-                                    
-                                    self._ShowHitPeriodicFileLimitMessage( query_name )
-                                    
-                                    stop_reason = 'hit periodic file limit without seeing any already-seen files!'
+                                    if total_already_in_urls_for_this_sync + num_urls_already_in_file_seed_cache > 0:
+                                        
+                                        # this sync produced some knowns, so it is likely we have stepped through a mix of old and tagged-late new files
+                                        # this is no reason to go crying to the user
+                                        
+                                        stop_reason = 'hit periodic file limit after seeing several already-seen files'
+                                        
+                                    else:
+                                        
+                                        # this page had all entirely new files
+                                        
+                                        self._ShowHitPeriodicFileLimitMessage( query_name )
+                                        
+                                        stop_reason = 'hit periodic file limit without seeing any already-seen files!'
+                                        
                                     
                                 
+                                can_search_for_more_files = False
+                                
+                                break
+                                
                             
-                            can_search_for_more_files = False
+                        
+                        if self._initial_file_limit is not None and self._periodic_file_limit is not None:
                             
-                            break
+                            # if the user has initial file sync of 5 but then normal sync of 100, we don't want to keep stomping through to older files on any subsequent normal sync
+                            # therefore, if we started this normal sync with fewer than normal sync files, we won't tolerate more than initial sync number of already in db
+                            if not this_is_initial_sync and num_file_seeds_at_start < self._periodic_file_limit and total_already_in_urls_for_this_sync >= self._initial_file_limit:
+                                
+                                stop_reason = 'believe I caught up with initial sync'
+                                
+                                can_search_for_more_files = False
+                                
+                                # since most initial file limits will be > 5, this will likely be superceded immediately by the WE_HIT_OLD_GROUND_THRESHOLD bit in a sec, but whatever
+                                
+                                break
+                                
                             
                         
                     

hydrus/client/importing/ClientImportWatchers.py

@@ -1230,7 +1230,14 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
                 
             elif not HydrusData.TimeHasPassed( self._no_work_until ):
                 
-                text = '{} - next check {}'.format( self._no_work_until_reason, ClientData.TimestampToPrettyTimeDelta( max( self._no_work_until, self._next_check_time ) ) )
+                if self._next_check_time is None:
+                    
+                    text = '{} - working again {}'.format( self._no_work_until_reason, ClientData.TimestampToPrettyTimeDelta( self._no_work_until ) )
+                    
+                else:
+                    
+                    text = '{} - next check {}'.format( self._no_work_until_reason, ClientData.TimestampToPrettyTimeDelta( max( self._no_work_until, self._next_check_time ) ) )
+                    
                 
                 return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_DEFERRED, text )
                 
@@ -1242,7 +1249,7 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
                     
                 else:
                     
-                    if HydrusData.TimeHasPassed( self._next_check_time ):
+                    if self._next_check_time is None or HydrusData.TimeHasPassed( self._next_check_time ):
                         
                         return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_PENDING, 'pending' )
                         
@@ -1271,7 +1278,14 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
                 
             elif not HydrusData.TimeHasPassed( self._no_work_until ):
                 
-                no_work_text = '{} - next check {}'.format( self._no_work_until_reason, ClientData.TimestampToPrettyTimeDelta( max( self._no_work_until, self._next_check_time ) ) )
+                if self._next_check_time is None:
+                    
+                    no_work_text = '{} - working again {}'.format( self._no_work_until_reason, ClientData.TimestampToPrettyTimeDelta( self._no_work_until ) )
+                    
+                else:
+                    
+                    no_work_text = '{} - next check {}'.format( self._no_work_until_reason, ClientData.TimestampToPrettyTimeDelta( max( self._no_work_until, self._next_check_time ) ) )
+                    
                 
                 file_status = no_work_text
                 watcher_status = no_work_text

hydrus/client/media/ClientMedia.py

@@ -2945,7 +2945,7 @@ class MediaSort( HydrusSerialisable.SerialisableBase ):
             
         else:
             
-            return ( 'ascending', 'descending', CC.SORT_BY_INCIDENCE_DESC )
+            return ( 'ascending', 'descending', CC.SORT_DESC )
             
         
     

hydrus/client/metadata/ClientTagSorting.py

@@ -1,8 +1,88 @@
+from hydrus.core import HydrusSerialisable
 from hydrus.core import HydrusTags
 
 from hydrus.client import ClientConstants as CC
 
-def SortTags( sort_by, list_of_tag_items, tag_items_to_count = None, item_to_tag_key_wrapper = None ):
+SORT_BY_HUMAN_TAG = 0
+SORT_BY_HUMAN_SUBTAG = 1
+SORT_BY_COUNT = 2
+
+GROUP_BY_NOTHING = 0
+GROUP_BY_NAMESPACE = 1
+
+sort_type_str_lookup = {
+    SORT_BY_HUMAN_TAG : 'sort by tag',
+    SORT_BY_HUMAN_SUBTAG : 'sort by subtag',
+    SORT_BY_COUNT : 'sort by count'
+}
+
+class TagSort( HydrusSerialisable.SerialisableBase ):
+    
+    SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_TAG_SORT
+    SERIALISABLE_NAME = 'Tag Sort'
+    SERIALISABLE_VERSION = 1
+    
+    def __init__( self, sort_type = None, sort_order = None, use_siblings = None, group_by = None ):
+        
+        if sort_type is None:
+            
+            sort_type = SORT_BY_HUMAN_TAG
+            
+        
+        if sort_order is None:
+            
+            sort_order = CC.SORT_ASC
+            
+        
+        if use_siblings is None:
+            
+            use_siblings = True
+            
+        
+        if group_by is None:
+            
+            group_by = GROUP_BY_NOTHING
+            
+        
+        self.sort_type = sort_type
+        self.sort_order = sort_order
+        self.use_siblings = use_siblings
+        self.group_by = group_by
+        
+    
+    def _GetSerialisableInfo( self ):
+        
+        return ( self.sort_type, self.sort_order, self.use_siblings, self.group_by )
+        
+    
+    def _InitialiseFromSerialisableInfo( self, serialisable_info ):
+        
+        ( self.sort_type, self.sort_order, self.use_siblings, self.group_by ) = serialisable_info
+        
+    
+    def ToString( self ):
+        
+        return '{} {}{}'.format(
+            sort_type_str_lookup[ self.sort_type ],
+            'asc' if self.sort_order == CC.SORT_ASC else 'desc',
+            ' namespace' if self.group_by == GROUP_BY_NAMESPACE else ''
+        )
+        
+    
+    @staticmethod
+    def STATICGetTextASCDefault() -> "TagSort":
+        
+        return TagSort(
+            sort_type = SORT_BY_HUMAN_TAG,
+            sort_order = CC.SORT_ASC,
+            use_siblings = True,
+            group_by = GROUP_BY_NOTHING
+        )
+        
+    
+HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_TAG_SORT ] = TagSort
+
+def SortTags( tag_sort: TagSort, list_of_tag_items, tag_items_to_count = None, item_to_tag_key_wrapper = None, item_to_sibling_key_wrapper = None ):
     
     def lexicographic_key( tag ):
         
@@ -72,16 +152,16 @@ def SortTags( sort_by, list_of_tag_items, tag_items_to_count = None, item_to_tag
     
     sorts_to_do = []
     
-    if sort_by in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
+    if tag_sort.sort_type == SORT_BY_COUNT:
         
         # let's establish a-z here for equal incidence values later
-        if sort_by in ( CC.SORT_BY_INCIDENCE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_ASC ):
+        if tag_sort.sort_order == CC.SORT_ASC:
             
             sorts_to_do.append( ( lexicographic_key, True ) )
             
             reverse = False
             
-        elif sort_by in ( CC.SORT_BY_INCIDENCE_DESC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
+        else:
             
             sorts_to_do.append( ( lexicographic_key, False ) )
             
@@ -90,15 +170,15 @@ def SortTags( sort_by, list_of_tag_items, tag_items_to_count = None, item_to_tag
         
         sorts_to_do.append( ( incidence_key, reverse ) )
         
-        if sort_by in ( CC.SORT_BY_INCIDENCE_NAMESPACE_ASC, CC.SORT_BY_INCIDENCE_NAMESPACE_DESC ):
+        if tag_sort.group_by == GROUP_BY_NAMESPACE:
             
             # python list sort is stable, so lets now sort again
             
-            if sort_by == CC.SORT_BY_INCIDENCE_NAMESPACE_ASC:
+            if tag_sort.sort_order == CC.SORT_ASC:
                 
                 reverse = True
                 
-            elif sort_by == CC.SORT_BY_INCIDENCE_NAMESPACE_DESC:
+            else:
                 
                 reverse = False
                 
@@ -108,27 +188,30 @@ def SortTags( sort_by, list_of_tag_items, tag_items_to_count = None, item_to_tag
         
     else:
         
-        if sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_DESC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC, CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC ):
-            
-            reverse = True
-            
-        elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC ):
+        if tag_sort.sort_order == CC.SORT_ASC:
             
             reverse = False
             
+        else:
+            
+            reverse = True
+            
         
-        if sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_NAMESPACE_DESC ):
-            
-            key = namespace_lexicographic_key
-            
-        elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_ASC, CC.SORT_BY_LEXICOGRAPHIC_DESC ):
-            
-            key = lexicographic_key
-            
-        elif sort_by in ( CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_ASC, CC.SORT_BY_LEXICOGRAPHIC_IGNORE_NAMESPACE_DESC ):
+        if tag_sort.sort_type == SORT_BY_HUMAN_SUBTAG:
             
             key = subtag_lexicographic_key
             
+        else:
+            
+            if tag_sort.group_by == GROUP_BY_NAMESPACE:
+                
+                key = namespace_lexicographic_key
+                
+            else:
+                
+                key = lexicographic_key
+                
+            
         
         sorts_to_do.append( ( key, reverse ) )
         
@@ -137,9 +220,13 @@ def SortTags( sort_by, list_of_tag_items, tag_items_to_count = None, item_to_tag
         
         key_to_use = key
         
-        if key_to_use != incidence_key: # other keys use tag, incidence uses tag item
+        if key_to_use is not incidence_key: # other keys use tag, incidence uses tag item
             
-            if item_to_tag_key_wrapper is not None:
+            if tag_sort.use_siblings and item_to_sibling_key_wrapper is not None:
+                
+                key_to_use = lambda item: key( item_to_sibling_key_wrapper( item ) )
+                
+            elif item_to_tag_key_wrapper is not None:
                 
                 key_to_use = lambda item: key( item_to_tag_key_wrapper( item ) )
                 
@@ -147,4 +234,4 @@ def SortTags( sort_by, list_of_tag_items, tag_items_to_count = None, item_to_tag
         
         list_of_tag_items.sort( key = key_to_use, reverse = reverse )
         
-    
+    

hydrus/client/networking/ClientNetworkingJobs.py

@@ -76,6 +76,10 @@ def ConvertStatusCodeAndDataIntoExceptionInfo( status_code, data, is_hydrus_serv
         
         eclass = HydrusExceptions.ConflictException
         
+    elif status_code == 416:
+        
+        eclass = HydrusExceptions.RangeNotSatisfiableException
+        
     elif status_code == 419:
         
         eclass = HydrusExceptions.SessionException

hydrus/core/HydrusConstants.py

@@ -70,7 +70,7 @@ options = {}
 # Misc
 
 NETWORK_VERSION = 19
-SOFTWARE_VERSION = 431
+SOFTWARE_VERSION = 432
 CLIENT_API_VERSION = 15
 
 SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )
@@ -139,6 +139,7 @@ CONTENT_TYPE_TIMESTAMP = 16
 CONTENT_TYPE_TITLE = 17
 CONTENT_TYPE_NOTES = 18
 CONTENT_TYPE_FILE_VIEWING_STATS = 19
+CONTENT_TYPE_TAG = 20
 
 content_type_string_lookup = {}
 

hydrus/core/HydrusDBModule.py

@@ -98,3 +98,10 @@ class HydrusDBModule( object ):
         raise NotImplementedError()
         
     
+    def GetTablesAndColumnsThatUseDefinitions( self, content_type: int ) -> typing.List[ typing.Tuple[ str, str ] ]:
+        
+        # could also do another one of these for orphan tables that have service id in the name.
+        
+        raise NotImplementedError()
+        
+    

hydrus/core/HydrusData.py

@@ -108,30 +108,46 @@ def ConvertIntToPrettyOrdinalString( num: int ):
         return 'unknown position'
         
     
-    remainder = abs( num ) % 10
+    tens = abs( num ) // 10
     
-    if remainder == 1:
+    if tens == 1:
         
-        ordinal = 'st'
-        
-    elif remainder == 2:
-        
-        ordinal = 'nd'
-        
-    elif remainder == 3:
-        
-        ordinal = 'rd'
+        ordinal = 'th'
         
     else:
         
-        ordinal = 'th'
+        remainder = abs( num ) % 10
+        
+        if remainder == 1:
+            
+            ordinal = 'st'
+            
+        elif remainder == 2:
+            
+            ordinal = 'nd'
+            
+        elif remainder == 3:
+            
+            ordinal = 'rd'
+            
+        else:
+            
+            ordinal = 'th'
+            
         
     
     s = '{}{}'.format( ToHumanInt( abs( num ) ), ordinal )
     
     if num < 0:
         
-        s = '{} from last'.format( s )
+        if num == -1:
+            
+            s = 'last'
+            
+        else:
+            
+            s = '{} from last'.format( s )
+            
         
     
     return s

hydrus/core/HydrusExceptions.py

@@ -88,6 +88,7 @@ class NotFoundException( NetworkException ): pass
 class NotModifiedException( NetworkException ): pass
 class BadRequestException( NetworkException ): pass
 class ConflictException( NetworkException ): pass
+class RangeNotSatisfiableException( NetworkException ): pass
 class MissingCredentialsException( NetworkException ): pass
 class DoesNotSupportCORSException( NetworkException ): pass
 class InsufficientCredentialsException( NetworkException ): pass

hydrus/core/HydrusSerialisable.py

@@ -119,6 +119,8 @@ SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER_SESSION_CONTAINER = 96
 SERIALISABLE_TYPE_NETWORK_BANDWIDTH_MANAGER_TRACKER_CONTAINER = 97
 SERIALISABLE_TYPE_SIDECAR_EXPORTER = 98
 SERIALISABLE_TYPE_STRING_SORTER = 99
+SERIALISABLE_TYPE_STRING_SLICER = 100
+SERIALISABLE_TYPE_TAG_SORT = 101
 
 SERIALISABLE_TYPES_TO_OBJECT_TYPES = {}
 

hydrus/core/HydrusServerResources.py

@@ -4,9 +4,10 @@ import traceback
 
 from twisted.internet import reactor, defer
 from twisted.internet.threads import deferToThread
+from twisted.python.failure import Failure
 from twisted.web.server import NOT_DONE_YET
 from twisted.web.resource import Resource
-from twisted.web.static import File as FileResource, NoRangeStaticProducer
+from twisted.web.static import File as FileResource, NoRangeStaticProducer, SingleRangeStaticProducer, MultipleRangeStaticProducer
 
 from hydrus.core import HydrusConstants as HC
 from hydrus.core import HydrusData
@@ -355,187 +356,6 @@ class HydrusResource( Resource ):
         return request
         
     
-    def _callbackEstablishAccountFromHeader( self, request ):
-        
-        return request
-        
-    
-    def _callbackEstablishAccountFromArgs( self, request ):
-        
-        return request
-        
-    
-    def _callbackParseGETArgs( self, request ):
-        
-        return request
-        
-    
-    def _callbackParsePOSTArgs( self, request ):
-        
-        return request
-        
-    
-    def _checkService( self, request ):
-        
-        return request
-        
-    
-    def _checkUserAgent( self, request ):
-        
-        request.is_hydrus_user_agent = False
-        
-        if request.requestHeaders.hasHeader( 'User-Agent' ):
-            
-            user_agent_texts = request.requestHeaders.getRawHeaders( 'User-Agent' )
-            
-            user_agent_text = user_agent_texts[0]
-            
-            try:
-                
-                user_agents = user_agent_text.split( ' ' )
-                
-            except:
-                
-                return # crazy user agent string, so just assume not a hydrus client
-                
-            
-            for user_agent in user_agents:
-                
-                if '/' in user_agent:
-                    
-                    ( client, network_version ) = user_agent.split( '/', 1 )
-                    
-                    if client == 'hydrus':
-                        
-                        request.is_hydrus_user_agent = True
-                        
-                        network_version = int( network_version )
-                        
-                        if network_version == HC.NETWORK_VERSION:
-                            
-                            return
-                            
-                        else:
-                            
-                            if network_version < HC.NETWORK_VERSION: message = 'Your client is out of date; please download the latest release.'
-                            else: message = 'This server is out of date; please ask its admin to update to the latest release.'
-                            
-                            raise HydrusExceptions.NetworkVersionException( 'Network version mismatch! This server\'s network version is ' + str( HC.NETWORK_VERSION ) + ', whereas your client\'s is ' + str( network_version ) + '! ' + message )
-                            
-                        
-                    
-                
-            
-        
-    
-    def _callbackRenderResponseContext( self, request ):
-        
-        self._CleanUpTempFile( request )
-        
-        if request.channel is None:
-            
-            # Connection was lost, it seems.
-            # no need for request.finish
-            
-            return
-            
-        
-        if request.requestHeaders.hasHeader( 'Origin' ):
-            
-            if self._service.SupportsCORS():
-                
-                request.setHeader( 'Access-Control-Allow-Origin', '*' )
-                
-            
-        
-        response_context = request.hydrus_response_context
-        
-        status_code = response_context.GetStatusCode()
-        
-        request.setResponseCode( status_code )
-        
-        for ( k, v, kwargs ) in response_context.GetCookies():
-            
-            request.addCookie( k, v, **kwargs )
-            
-        
-        do_finish = True
-        
-        if response_context.HasPath():
-            
-            path = response_context.GetPath()
-            
-            size = os.path.getsize( path )
-            
-            mime = response_context.GetMime()
-            
-            content_type = HC.mime_mimetype_string_lookup[ mime ]
-            
-            content_length = size
-            
-            ( base, filename ) = os.path.split( path )
-            
-            content_disposition = 'inline; filename="' + filename + '"'
-            
-            request.setHeader( 'Content-Type', str( content_type ) )
-            request.setHeader( 'Content-Length', str( content_length ) )
-            request.setHeader( 'Content-Disposition', str( content_disposition ) )
-            
-            request.setHeader( 'Expires', time.strftime( '%a, %d %b %Y %H:%M:%S GMT', time.gmtime( time.time() + 86400 * 365 ) ) )
-            request.setHeader( 'Cache-Control', 'max-age={}'.format( 86400 * 365 ) )
-            
-            fileObject = open( path, 'rb' )
-            
-            producer = NoRangeStaticProducer( request, fileObject )
-            
-            producer.start()
-            
-            do_finish = False
-            
-        elif response_context.HasBody():
-            
-            mime = response_context.GetMime()
-            
-            body_bytes = response_context.GetBodyBytes()
-            
-            content_type = HC.mime_mimetype_string_lookup[ mime ]
-            
-            content_length = len( body_bytes )
-            
-            content_disposition = 'inline'
-            
-            request.setHeader( 'Content-Type', content_type )
-            request.setHeader( 'Content-Length', str( content_length ) )
-            request.setHeader( 'Content-Disposition', content_disposition )
-            
-            request.write( body_bytes )
-            
-        else:
-            
-            content_length = 0
-            
-            if status_code != 204: # 204 is No Content
-                
-                request.setHeader( 'Content-Length', str( content_length ) )
-                
-            
-        
-        self._reportDataUsed( request, content_length )
-        self._reportRequestUsed( request )
-        
-        if do_finish:
-            
-            request.finish()
-            
-        
-    
-    def _profileJob( self, call, request ):
-        
-        HydrusData.Profile( 'client api {}'.format( request.path ), 'request.result_lmao = call( request )', globals(), locals(), min_duration_ms = 3, show_summary = True )
-        
-        return request.result_lmao
-        
-    
     def _callbackDoGETJob( self, request ):
         
         def wrap_thread_result( response_context ):
@@ -605,6 +425,233 @@ class HydrusResource( Resource ):
         return d
         
     
+    def _callbackEstablishAccountFromHeader( self, request ):
+        
+        return request
+        
+    
+    def _callbackEstablishAccountFromArgs( self, request ):
+        
+        return request
+        
+    
+    def _callbackParseGETArgs( self, request ):
+        
+        return request
+        
+    
+    def _callbackParsePOSTArgs( self, request ):
+        
+        return request
+        
+    
+    def _callbackRenderResponseContext( self, request ):
+        
+        self._CleanUpTempFile( request )
+        
+        if request.channel is None:
+            
+            # Connection was lost, it seems.
+            # no need for request.finish
+            
+            return
+            
+        
+        if request.requestHeaders.hasHeader( 'Origin' ):
+            
+            if self._service.SupportsCORS():
+                
+                request.setHeader( 'Access-Control-Allow-Origin', '*' )
+                
+            
+        
+        response_context = request.hydrus_response_context
+        
+        if response_context.HasPath():
+            
+            path = response_context.GetPath()
+            
+            size = os.path.getsize( path )
+            
+            offset_and_block_size_pairs = self._parseRangeHeader( request, size )
+            
+        else:
+            
+            offset_and_block_size_pairs = []
+            
+        
+        status_code = response_context.GetStatusCode()
+        
+        if status_code == 200 and response_context.HasPath() and len( offset_and_block_size_pairs ) > 0:
+            
+            status_code = 206
+            
+        
+        request.setResponseCode( status_code )
+        
+        for ( k, v, kwargs ) in response_context.GetCookies():
+            
+            request.addCookie( k, v, **kwargs )
+            
+        
+        do_finish = True
+        
+        if response_context.HasPath():
+            
+            path = response_context.GetPath()
+            
+            size = os.path.getsize( path )
+            
+            mime = response_context.GetMime()
+            
+            content_type = HC.mime_mimetype_string_lookup[ mime ]
+            
+            ( base, filename ) = os.path.split( path )
+            
+            fileObject = open( path, 'rb' )
+            
+            content_disposition = 'inline; filename="' + filename + '"'
+            
+            request.setHeader( 'Content-Disposition', str( content_disposition ) )
+            
+            request.setHeader( 'Expires', time.strftime( '%a, %d %b %Y %H:%M:%S GMT', time.gmtime( time.time() + 86400 * 365 ) ) )
+            request.setHeader( 'Cache-Control', 'max-age={}'.format( 86400 * 365 ) )
+            
+            if len( offset_and_block_size_pairs ) <= 1:
+                
+                request.setHeader( 'Content-Type', str( content_type ) )
+                
+                if len( offset_and_block_size_pairs ) == 0:
+                    
+                    content_length = size
+                    
+                    request.setHeader( 'Content-Length', str( content_length ) )
+                    
+                    producer = NoRangeStaticProducer( request, fileObject )
+                    
+                elif len( offset_and_block_size_pairs ) == 1:
+                    
+                    [ ( range_start, range_end, offset, block_size ) ] = offset_and_block_size_pairs
+                    
+                    content_length = block_size
+                    
+                    request.setHeader( 'Accept-Ranges', 'bytes' )
+                    request.setHeader( 'Content-Range', 'bytes {}-{}/{}'.format( offset, range_end, size ) )
+                    request.setHeader( 'Content-Length', str( content_length ) )
+                    
+                    producer = SingleRangeStaticProducer( request, fileObject, offset, block_size )
+                    
+                
+            else:
+                
+                # hey, what a surprise, an http data transmission standard turned out to be a massive PITA
+                # MultipleRangeStaticProducer is the lad to use, but you have to figure out your own separation bits, which have even more finicky rules. more than I can deal with with the current time I have
+                # if/when you want to do this, check out the FileResource, it does it in its internal gubbins
+                
+                raise HydrusExceptions.RangeNotSatisfiableException( 'Can only support Single Range requests at the moment!' )
+                
+            
+            producer.start()
+            
+            do_finish = False
+            
+        elif response_context.HasBody():
+            
+            mime = response_context.GetMime()
+            
+            body_bytes = response_context.GetBodyBytes()
+            
+            content_type = HC.mime_mimetype_string_lookup[ mime ]
+            
+            content_length = len( body_bytes )
+            
+            content_disposition = 'inline'
+            
+            request.setHeader( 'Content-Type', content_type )
+            request.setHeader( 'Content-Length', str( content_length ) )
+            request.setHeader( 'Content-Disposition', content_disposition )
+            
+            request.write( body_bytes )
+            
+        else:
+            
+            content_length = 0
+            
+            if status_code != 204: # 204 is No Content
+                
+                request.setHeader( 'Content-Length', str( content_length ) )
+                
+            
+        
+        self._reportDataUsed( request, content_length )
+        self._reportRequestUsed( request )
+        
+        if do_finish:
+            
+            request.finish()
+            
+        
+    
+    def _checkService( self, request ):
+        
+        return request
+        
+    
+    def _checkUserAgent( self, request ):
+        
+        request.is_hydrus_user_agent = False
+        
+        if request.requestHeaders.hasHeader( 'User-Agent' ):
+            
+            user_agent_texts = request.requestHeaders.getRawHeaders( 'User-Agent' )
+            
+            user_agent_text = user_agent_texts[0]
+            
+            try:
+                
+                user_agents = user_agent_text.split( ' ' )
+                
+            except:
+                
+                return # crazy user agent string, so just assume not a hydrus client
+                
+            
+            for user_agent in user_agents:
+                
+                if '/' in user_agent:
+                    
+                    ( client, network_version ) = user_agent.split( '/', 1 )
+                    
+                    if client == 'hydrus':
+                        
+                        request.is_hydrus_user_agent = True
+                        
+                        network_version = int( network_version )
+                        
+                        if network_version == HC.NETWORK_VERSION:
+                            
+                            return
+                            
+                        else:
+                            
+                            if network_version < HC.NETWORK_VERSION: message = 'Your client is out of date; please download the latest release.'
+                            else: message = 'This server is out of date; please ask its admin to update to the latest release.'
+                            
+                            raise HydrusExceptions.NetworkVersionException( 'Network version mismatch! This server\'s network version is ' + str( HC.NETWORK_VERSION ) + ', whereas your client\'s is ' + str( network_version ) + '! ' + message )
+                            
+                        
+                    
+                
+            
+        
+    
+    def _profileJob( self, call, request ):
+        
+        HydrusData.Profile( 'client api {}'.format( request.path ), 'request.result_lmao = call( request )', globals(), locals(), min_duration_ms = 3, show_summary = True )
+        
+        return request.result_lmao
+        
+    
     def _DecompressionBombsOK( self, request ):
         
         return False
@@ -615,85 +662,91 @@ class HydrusResource( Resource ):
         request_deferred.cancel()
         
     
-    def _errbackHandleEmergencyError( self, failure, request ):
-        
-        try: self._CleanUpTempFile( request )
-        except: pass
-        
-        try: HydrusData.DebugPrint( failure.getTraceback() )
-        except: pass
-        
-        if request.channel is not None:
-            
-            try: request.setResponseCode( 500 )
-            except: pass
-            
-            try: request.write( failure.getTraceback() )
-            except: pass
-            
-        
-        if not request.finished:
-            
-            try: request.finish()
-            except: pass
-            
-        
-    
     def _errbackHandleProcessingError( self, failure, request ):
         
-        self._CleanUpTempFile( request )
-        
-        default_mime = HC.TEXT_HTML
-        default_encoding = str
-        
-        if failure.type == HydrusExceptions.BadRequestException:
+        try:
             
-            response_context = ResponseContext( 400, mime = default_mime, body = default_encoding( failure.value ) )
+            try: self._CleanUpTempFile( request )
+            except: pass
             
-        elif failure.type in ( HydrusExceptions.MissingCredentialsException, HydrusExceptions.DoesNotSupportCORSException ):
+            default_mime = HC.TEXT_HTML
+            default_encoding = str
             
-            response_context = ResponseContext( 401, mime = default_mime, body = default_encoding( failure.value ) )
+            if failure.type == HydrusExceptions.BadRequestException:
+                
+                response_context = ResponseContext( 400, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type in ( HydrusExceptions.MissingCredentialsException, HydrusExceptions.DoesNotSupportCORSException ):
+                
+                response_context = ResponseContext( 401, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type == HydrusExceptions.InsufficientCredentialsException:
+                
+                response_context = ResponseContext( 403, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type in ( HydrusExceptions.NotFoundException, HydrusExceptions.DataMissing, HydrusExceptions.FileMissingException ):
+                
+                response_context = ResponseContext( 404, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type == HydrusExceptions.ConflictException:
+                
+                response_context = ResponseContext( 409, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type == HydrusExceptions.RangeNotSatisfiableException:
+                
+                response_context = ResponseContext( 416, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type == HydrusExceptions.SessionException:
+                
+                response_context = ResponseContext( 419, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type == HydrusExceptions.NetworkVersionException:
+                
+                response_context = ResponseContext( 426, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type == HydrusExceptions.ServerBusyException:
+                
+                response_context = ResponseContext( 503, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type == HydrusExceptions.BandwidthException:
+                
+                response_context = ResponseContext( 509, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            elif failure.type == HydrusExceptions.ServerException:
+                
+                response_context = ResponseContext( 500, mime = default_mime, body = default_encoding( failure.value ) )
+                
+            else:
+                
+                HydrusData.DebugPrint( failure.getTraceback() )
+                
+                response_context = ResponseContext( 500, mime = default_mime, body = default_encoding( 'The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com:' + os.linesep * 2 + failure.getTraceback() ) )
+                
             
-        elif failure.type == HydrusExceptions.InsufficientCredentialsException:
+            request.hydrus_response_context = response_context
             
-            response_context = ResponseContext( 403, mime = default_mime, body = default_encoding( failure.value ) )
+            self._callbackRenderResponseContext( request )
             
-        elif failure.type in ( HydrusExceptions.NotFoundException, HydrusExceptions.DataMissing, HydrusExceptions.FileMissingException ):
+        except:
             
-            response_context = ResponseContext( 404, mime = default_mime, body = default_encoding( failure.value ) )
+            try: HydrusData.DebugPrint( failure.getTraceback() )
+            except: pass
             
-        elif failure.type == HydrusExceptions.ConflictException:
+            if hasattr( request, 'channel' ) and request.channel is not None:
+                
+                try: request.setResponseCode( 500 )
+                except: pass
+                
+                try: request.write( failure.getTraceback() )
+                except: pass
+                
             
-            response_context = ResponseContext( 409, mime = default_mime, body = default_encoding( failure.value ) )
+            if not request.finished:
+                
+                try: request.finish()
+                except: pass
+                
             
-        elif failure.type == HydrusExceptions.SessionException:
-            
-            response_context = ResponseContext( 419, mime = default_mime, body = default_encoding( failure.value ) )
-            
-        elif failure.type == HydrusExceptions.NetworkVersionException:
-            
-            response_context = ResponseContext( 426, mime = default_mime, body = default_encoding( failure.value ) )
-            
-        elif failure.type == HydrusExceptions.ServerBusyException:
-            
-            response_context = ResponseContext( 503, mime = default_mime, body = default_encoding( failure.value ) )
-            
-        elif failure.type == HydrusExceptions.BandwidthException:
-            
-            response_context = ResponseContext( 509, mime = default_mime, body = default_encoding( failure.value ) )
-            
-        elif failure.type == HydrusExceptions.ServerException:
-            
-            response_context = ResponseContext( 500, mime = default_mime, body = default_encoding( failure.value ) )
-            
-        else:
-            
-            HydrusData.DebugPrint( failure.getTraceback() )
-            
-            response_context = ResponseContext( 500, mime = default_mime, body = default_encoding( 'The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com:' + os.linesep * 2 + failure.getTraceback() ) )
-            
-        
-        request.hydrus_response_context = response_context
         
         return request
         
@@ -728,6 +781,104 @@ class HydrusResource( Resource ):
         return access_key
         
     
+    def _parseRangeHeader( self, request, size ):
+        
+        offset_and_block_size_pairs = []
+        
+        if request.requestHeaders.hasHeader( 'Range' ):
+            
+            range_headers = request.requestHeaders.getRawHeaders( 'Range' )
+            
+            range_header = range_headers[0]
+            
+            if '=' not in range_header:
+                
+                raise HydrusExceptions.BadRequestException( 'Did not understand range header!' )
+                
+            
+            ( unit_gumpf, range_pairs_string ) = range_header.split( '=', 1 )
+            
+            if unit_gumpf != 'bytes':
+                
+                raise HydrusExceptions.RangeNotSatisfiableException( 'Do not support anything other than bytes in Range header!' )
+                
+            
+            range_pair_strings = range_pairs_string.split( ',' )
+            
+            if True in ( '-' not in range_pair_string for range_pair_string in range_pair_strings ):
+                
+                raise HydrusExceptions.RangeNotSatisfiableException( 'Did not understand the Range header\'s range pair(s)!' )
+                
+            
+            range_pairs = [ range_pair_string.strip().split( '-' ) for range_pair_string in range_pair_strings ]
+            
+            offset_and_block_size_pairs = []
+            
+            for ( range_start, range_end ) in range_pairs:
+                
+                if range_start == '':
+                    
+                    if range_end == '':
+                        
+                        raise HydrusExceptions.RangeNotSatisfiableException( 'Undefined Range header pair given!' )
+                        
+                    
+                    range_start = None
+                    
+                else:
+                    
+                    range_start = abs( int( range_start ) )
+                    
+                
+                if range_end == '':
+                    
+                    range_end = None
+                    
+                else:
+                    
+                    range_end = abs( int( range_end ) )
+                    
+                
+                if range_start is not None and range_end is not None and range_start > range_end:
+                    
+                    raise HydrusExceptions.RangeNotSatisfiableException( 'The Range header had an invalid pair!' )
+                    
+                
+                if range_start is None:
+                    
+                    offset = size - range_end
+                    block_size = range_end
+                    
+                elif range_end is None:
+                    
+                    offset = range_start
+                    block_size = size - range_start
+                    
+                else:
+                    
+                    if range_start > size:
+                        
+                        offset_and_block_size_pairs = []
+                        
+                        break
+                        
+                    
+                    if range_end > size:
+                        
+                        range_end = size - 1
+                        
+                    
+                    offset = range_start
+                    block_size = ( range_end + 1 ) - range_start
+                    
+                
+                offset_and_block_size_pairs.append( ( range_start, range_end, offset, block_size ) )
+                
+            
+        
+        return offset_and_block_size_pairs
+        
+    
     def _reportDataUsed( self, request, num_bytes ):
         
         self._service.ReportDataUsed( num_bytes )
@@ -827,11 +978,9 @@ class HydrusResource( Resource ):
         
         d.addCallback( self._callbackDoGETJob )
         
-        d.addErrback( self._errbackHandleProcessingError, request )
-        
         d.addCallback( self._callbackRenderResponseContext )
         
-        d.addErrback( self._errbackHandleEmergencyError, request )
+        d.addErrback( self._errbackHandleProcessingError, request )
         
         request.notifyFinish().addErrback( self._errbackDisconnected, d )
         
@@ -850,11 +999,9 @@ class HydrusResource( Resource ):
         
         d.addCallback( self._callbackDoOPTIONSJob )
         
-        d.addErrback( self._errbackHandleProcessingError, request )
-        
         d.addCallback( self._callbackRenderResponseContext )
         
-        d.addErrback( self._errbackHandleEmergencyError, request )
+        d.addErrback( self._errbackHandleProcessingError, request )
         
         request.notifyFinish().addErrback( self._errbackDisconnected, d )
         
@@ -881,11 +1028,9 @@ class HydrusResource( Resource ):
         
         d.addCallback( self._callbackDoPOSTJob )
         
-        d.addErrback( self._errbackHandleProcessingError, request )
-        
         d.addCallback( self._callbackRenderResponseContext )
         
-        d.addErrback( self._errbackHandleEmergencyError, request )
+        d.addErrback( self._errbackHandleProcessingError, request )
         
         request.notifyFinish().addErrback( self._errbackDisconnected, d )
         

hydrus/test/TestClientAPI.py

@@ -2106,6 +2106,106 @@ class TestClientAPI( unittest.TestCase ):
         
         self.assertEqual( hashlib.sha256( data ).digest(), hash )
         
+        # range request
+        
+        path = '/get_files/file?file_id={}'.format( 1 )
+        
+        partial_headers = dict( headers )
+        partial_headers[ 'Range' ] = 'bytes=100-199'
+        
+        connection.request( 'GET', path, headers = partial_headers )
+        
+        response = connection.getresponse()
+        
+        data = response.read()
+        
+        self.assertEqual( response.status, 206 )
+        
+        with open( file_path, 'rb' ) as f:
+            
+            f.seek( 100 )
+            
+            actual_data = f.read( 100 )
+            
+        
+        self.assertEqual( data, actual_data )
+        
+        # n onwards range request
+        
+        path = '/get_files/file?file_id={}'.format( 1 )
+        
+        partial_headers = dict( headers )
+        partial_headers[ 'Range' ] = 'bytes=100-'
+        
+        connection.request( 'GET', path, headers = partial_headers )
+        
+        response = connection.getresponse()
+        
+        data = response.read()
+        
+        self.assertEqual( response.status, 206 )
+        
+        with open( file_path, 'rb' ) as f:
+            
+            f.seek( 100 )
+            
+            actual_data = f.read()
+            
+        
+        self.assertEqual( data, actual_data )
+        
+        # last n onwards range request
+        
+        path = '/get_files/file?file_id={}'.format( 1 )
+        
+        partial_headers = dict( headers )
+        partial_headers[ 'Range' ] = 'bytes=-100'
+        
+        connection.request( 'GET', path, headers = partial_headers )
+        
+        response = connection.getresponse()
+        
+        data = response.read()
+        
+        self.assertEqual( response.status, 206 )
+        
+        with open( file_path, 'rb' ) as f:
+            
+            actual_data = f.read()[-100:]
+            
+        
+        self.assertEqual( data, actual_data )
+        
+        # invalid range request
+        
+        path = '/get_files/file?file_id={}'.format( 1 )
+        
+        partial_headers = dict( headers )
+        partial_headers[ 'Range' ] = 'bytes=200-199'
+        
+        connection.request( 'GET', path, headers = partial_headers )
+        
+        response = connection.getresponse()
+        
+        data = response.read()
+        
+        self.assertEqual( response.status, 416 )
+        
+        # multi range request, not currently supported
+        
+        path = '/get_files/file?file_id={}'.format( 1 )
+        
+        partial_headers = dict( headers )
+        partial_headers[ 'Range' ] = 'bytes=100-199,300-399'
+        
+        connection.request( 'GET', path, headers = partial_headers )
+        
+        response = connection.getresponse()
+        
+        data = response.read()
+        
+        self.assertEqual( response.status, 416 )
+        
         #
         
         path = '/get_files/thumbnail?file_id={}'.format( 1 )

hydrus/test/TestClientDB.py

@@ -1211,7 +1211,7 @@ class TestClientDB( unittest.TestCase ):
         
         result = self._read( 'serialisable_simple', 'pixiv_account' )
         
-        self.assertTrue( result, ( pixiv_id, password ) )
+        self.assertEqual( result, [ pixiv_id, password ] )
         
     
     def test_services( self ):

hydrus/test/TestClientNetworking.py

@@ -641,8 +641,8 @@ class TestNetworkingJob( unittest.TestCase ):
                 
                 tracker = bm.GetTracker( ClientNetworkingContexts.GLOBAL_NETWORK_CONTEXT )
                 
-                self.assertTrue( tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 1 )
-                self.assertTrue( tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 256 )
+                self.assertEqual( tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 1 )
+                self.assertEqual( tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 256 )
                 
             
         
@@ -682,7 +682,7 @@ class TestNetworkingJob( unittest.TestCase ):
                 
                 self.assertEqual( type( job.GetErrorException() ), HydrusExceptions.ServerException )
                 
-                self.assertTrue( job.GetErrorText(), BAD_RESPONSE )
+                self.assertEqual( job.GetErrorText(), BAD_RESPONSE.decode( 'ascii' ) )
                 
                 self.assertEqual( job.GetStatus(), ( '500 - Internal Server Error', 18, 18, None ) )
                 
@@ -837,7 +837,7 @@ class TestNetworkingJobHydrus( unittest.TestCase ):
                 
                 self.assertEqual( type( job.GetErrorException() ), HydrusExceptions.ServerException )
                 
-                self.assertTrue( job.GetErrorText(), BAD_RESPONSE )
+                self.assertEqual( job.GetErrorText(), BAD_RESPONSE.decode( 'ascii' ) )
                 
                 self.assertEqual( job.GetStatus(), ( '500 - Internal Server Error', 18, 18, None ) )
                 

hydrus/test/TestClientParsing.py

@@ -1,3 +1,4 @@
+import os
 import random
 import unittest
 
@@ -281,6 +282,140 @@ class TestStringMatch( unittest.TestCase ):
         self.assertTrue( re_string_match.Matches( 'abc123' ) )
         
     
+class TestStringSlicer( unittest.TestCase ):
+    
+    def test_basics( self ):
+        
+        a = 'a ' + os.urandom( 8 ).hex()
+        b = 'b ' + os.urandom( 8 ).hex()
+        c = 'c ' + os.urandom( 8 ).hex()
+        d = 'd ' + os.urandom( 8 ).hex()
+        e = 'e ' + os.urandom( 8 ).hex()
+        f = 'f ' + os.urandom( 8 ).hex()
+        g = 'g ' + os.urandom( 8 ).hex()
+        h = 'h ' + os.urandom( 8 ).hex()
+        i = 'i ' + os.urandom( 8 ).hex()
+        j = 'j ' + os.urandom( 8 ).hex()
+        
+        test_list = [ a, b, c, d, e, f, g, h, i, j ]
+        
+        #
+        
+        slicer = ClientParsing.StringSlicer( index_start = 0, index_end = 1 )
+        self.assertEqual( slicer.Slice( test_list ), [ a ] )
+        self.assertEqual( slicer.ToString(), 'selecting the 1st string' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = 3, index_end = 4 )
+        self.assertEqual( slicer.Slice( test_list ), [ d ] )
+        self.assertEqual( slicer.ToString(), 'selecting the 4th string' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = -3, index_end = -2 )
+        self.assertEqual( slicer.Slice( test_list ), [ h ] )
+        self.assertEqual( slicer.ToString(), 'selecting the 3rd from last string' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = -1 )
+        self.assertEqual( slicer.Slice( test_list ), [ j ] )
+        self.assertEqual( slicer.ToString(), 'selecting the last string' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = 15, index_end = 16 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting the 16th string' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = -15, index_end = -14 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting the 15th from last string' )
+        
+        #
+        
+        slicer = ClientParsing.StringSlicer( index_start = 0 )
+        self.assertEqual( slicer.Slice( test_list ), test_list )
+        self.assertEqual( slicer.ToString(), 'selecting the 1st string and onwards' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = 3 )
+        self.assertEqual( slicer.Slice( test_list ), [ d, e, f, g, h, i, j ] )
+        self.assertEqual( slicer.ToString(), 'selecting the 4th string and onwards' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = -3 )
+        self.assertEqual( slicer.Slice( test_list ), [ h, i, j ] )
+        self.assertEqual( slicer.ToString(), 'selecting the 3rd from last string and onwards' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = 15 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting the 16th string and onwards' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = -15 )
+        self.assertEqual( slicer.Slice( test_list ), test_list )
+        self.assertEqual( slicer.ToString(), 'selecting the 15th from last string and onwards' )
+        
+        #
+        
+        slicer = ClientParsing.StringSlicer( index_end = 0 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting nothing' )
+        
+        slicer = ClientParsing.StringSlicer( index_end = 3 )
+        self.assertEqual( slicer.Slice( test_list ), [ a, b, c ] )
+        self.assertEqual( slicer.ToString(), 'selecting up to and including the 3rd string' )
+        
+        slicer = ClientParsing.StringSlicer( index_end = -3 )
+        self.assertEqual( slicer.Slice( test_list ), [ a, b, c, d, e, f, g ] )
+        self.assertEqual( slicer.ToString(), 'selecting up to and including the 4th from last string' )
+        
+        slicer = ClientParsing.StringSlicer( index_end = 15 )
+        self.assertEqual( slicer.Slice( test_list ), test_list )
+        self.assertEqual( slicer.ToString(), 'selecting up to and including the 15th string' )
+        
+        slicer = ClientParsing.StringSlicer( index_end = -15 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting up to and including the 16th from last string' )
+        
+        #
+        
+        slicer = ClientParsing.StringSlicer( index_start = 0, index_end = 5 )
+        self.assertEqual( slicer.Slice( test_list ), [ a, b, c, d, e ] )
+        self.assertEqual( slicer.ToString(), 'selecting the 1st string up to and including the 5th string' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = 3, index_end = 5 )
+        self.assertEqual( slicer.Slice( test_list ), [ d, e ] )
+        self.assertEqual( slicer.ToString(), 'selecting the 4th string up to and including the 5th string' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = -5, index_end = -3 )
+        self.assertEqual( slicer.Slice( test_list ), [ f, g ] )
+        self.assertEqual( slicer.ToString(), 'selecting the 5th from last string up to and including the 4th from last string' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = 3, index_end = -3 )
+        self.assertEqual( slicer.Slice( test_list ), [ d, e, f, g ] )
+        self.assertEqual( slicer.ToString(), 'selecting the 4th string up to and including the 4th from last string' )
+        
+        #
+        
+        slicer = ClientParsing.StringSlicer( index_start = 3, index_end = 3 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting nothing' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = 5, index_end = 3 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting nothing' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = -3, index_end = -3 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting nothing' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = -3, index_end = -5 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting nothing' )
+        
+        #
+        
+        slicer = ClientParsing.StringSlicer( index_start = 15, index_end = 20 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting the 16th string up to and including the 20th string' )
+        
+        slicer = ClientParsing.StringSlicer( index_start = -15, index_end = -12 )
+        self.assertEqual( slicer.Slice( test_list ), [] )
+        self.assertEqual( slicer.ToString(), 'selecting the 15th from last string up to and including the 13th from last string' )
+        
+    
 class TestStringSorter( unittest.TestCase ):
     
     def test_basics( self ):
@@ -299,7 +434,7 @@ class TestStringSorter( unittest.TestCase ):
                 
                 random.shuffle( test_list )
                 
-                self.assertTrue( sorter.Sort( test_list ), correct )
+                self.assertEqual( sorter.Sort( test_list ), correct )
                 
             
         
@@ -356,15 +491,15 @@ class TestStringSplitter( unittest.TestCase ):
         
         splitter = ClientParsing.StringSplitter( separator = ', ' )
         
-        self.assertTrue( splitter.Split( '123' ), [ '123' ] )
-        self.assertTrue( splitter.Split( '1,2,3' ), [ '1,2,3' ] )
-        self.assertTrue( splitter.Split( '1, 2, 3' ), [ '1', '2', '3' ] )
+        self.assertEqual( splitter.Split( '123' ), [ '123' ] )
+        self.assertEqual( splitter.Split( '1,2,3' ), [ '1,2,3' ] )
+        self.assertEqual( splitter.Split( '1, 2, 3' ), [ '1', '2', '3' ] )
         
         splitter = ClientParsing.StringSplitter( separator = ', ', max_splits = 2 )
         
-        self.assertTrue( splitter.Split( '123' ), [ '123' ] )
-        self.assertTrue( splitter.Split( '1,2,3' ), [ '1,2,3' ] )
-        self.assertTrue( splitter.Split( '1, 2, 3, 4' ), [ '1', '2', '3,4' ] )
+        self.assertEqual( splitter.Split( '123' ), [ '123' ] )
+        self.assertEqual( splitter.Split( '1,2,3' ), [ '1,2,3' ] )
+        self.assertEqual( splitter.Split( '1, 2, 3, 4' ), [ '1', '2', '3, 4' ] )
         
     
 class TestStringProcessor( unittest.TestCase ):

hydrus/test/TestHydrusServer.py

@@ -172,6 +172,8 @@ class TestServer( unittest.TestCase ):
         
         self.assertEqual( response, EXAMPLE_FILE )
         
+        #
+        
         try: os.remove( path )
         except: pass