Version 184

client.pyw

@@ -27,8 +27,6 @@ try:
     from include import HydrusLogger
     import traceback
     
-    HydrusGlobals.instance = HC.HYDRUS_CLIENT
-    
     with HydrusLogger.HydrusLogger( 'client.log' ) as logger:
         
         try:

help/advanced.html

@@ -20,6 +20,8 @@
 			<p>In this case, selecting the 'title:cool pic*' predicate will return all three images in the same search, where you can conveniently give them some more-easily searched tags like 'series:cool pic' and 'page:1', 'page:2', 'page:3'.</p>
 			<h3>exclude deleted files</h3>
 			<p>In the client's options is a checkbox to exclude deleted files. It recurs pretty much anywhere you can import, under 'import file options'. If you select this, any file you ever deleted will be excluded from all future remote searches and import operations. This can stop you from importing/downloading and filtering out the same bad files several times over. The default is off. You may wish to have it set one way most of the time, but switch it the other just for one specific import or search.</p>
+			<h3>inputting non-english lanuages</h3>
+			<p>If you typically use an IME to input Japanese or another non-english language, you may have encountered problems entering into the autocomplete tag entry control in that you need Up/Down/Enter to navigate the IME, but the autocomplete steals those key presses away to navigate the list of results. To fix this, press Shift+Tab to temporarily disable the autocomplete's key event capture. The autocomplete text box will change colour to let you know it has released its normal key capture. Use your IME to get the text you want, then hit Shift+Tab again to restore the autocomplete to normal behaviour.</p>
 			<h3>tag censorship</h3>
 			<p>If you do not like a particular tag or namespace, you can easily hide it with <i>services->manage tag censorship</i>:</p>
 			<p><img src="tag_censorship.png" /></p>

help/changelog.html

@@ -8,6 +8,39 @@
 		<div class="content">
 			<h3>changelog</h3>
 			<ul>
+				<li><h3>version 184</h3></li>
+				<ul>
+					<li>added external client_files storage!</li>
+					<li>you can add external client_files folders in options->file storage locations, further giving them weight</li>
+					<li>a new daemon will incrementally rebalance your files (and recover orphaned subfolders!) over your different storage locations</li>
+					<li>you can also force a full rebalance from the new database->maintenance->rebalance file storage</li>
+					<li>simplified the maintenance and processing panel</li>
+					<li>the maintenance and processing panel controls now appropriately support being set to 'none'</li>
+					<li>the 'run jobs on idle?' question is now explicit on the maintenance and processing panel</li>
+					<li>deselecting 'run jobs on idle/shutdown' will now disable subordinate controls</li>
+					<li>hitting tab on the autocomplete control now triggers an immediate 'fetch results' call</li>
+					<li>added a checkbox to options->speed and memory to completely disable automatic autocomplete results fetching (i.e. if you want to manually control a/c result-fetching only with this new tab shortcut)</li>
+					<li>the new less-laggy autocomplete results fetch won't trigger if the latest query is shorter than the cached query (i.e. you won't get lag when hitting backspace a bunch of times on autocomplete)</li>
+					<li>hitting shift+tab on the autocomplete control now disables all other key event capture, letting you enter IME without your up/down/enter presses controlling the dropdown list of results</li>
+					<li>'waiting politely' time is now indicated with a small 'traffic light'-type circle control on all downloader pages</li>
+					<li>fixed some occasional 'I'll go sit in the top-left corner of the screen and not fix my position' hover window behaviour in Linux</li>
+					<li>fixed sibling predicate collapse for the 'read' autocomplete dropdown for database results</li>
+					<li>refactored how predicates are collapsed and sorted to be a bit more sensible</li>
+					<li>fixed maintenance of local booru data use even if local booru does not receive any requests</li>
+					<li>improved some service/content update error handling in repository sync</li>
+					<li>the system:rating value-entry dialog no longer lists its rating services in random order</li>
+					<li>'open externally' will no longer show for non-local thumbnails' right-click menus</li>
+					<li>fixed audio/pdf thumbnail display for local booru, although they will be full size for now</li>
+					<li>harmonised some hex-prefix folder generation code</li>
+					<li>improved some file and directory copying code</li>
+					<li>improved upload pending popup message cleanup</li>
+					<li>added help_dir to hydrusconstants</li>
+					<li>miscellaneous refactoring</li>
+					<li>misc cleanup</li>
+					<li>extracted server-specific services and server resources from the common import path to the server import path</li>
+					<li>a bit more misc server refactoring</li>
+					<li>updated a couple of bits of help</li>
+				</ul>
 				<li><h3>version 183</h3></li>
 				<ul>
 					<li>added swf thumbnail support--it works ok for most swfs!</li>

help/server.html

@@ -53,7 +53,7 @@
 			<p>All of a server's files and options are stored in its accompanying .db file and respective subdirectories, which are created on first startup (just like with the client). To backup or restore, you have two options:</p>
 			<ul>
 				<li>Shut down the server, copy the database files and directories, then restart it. This is the only way, currently, to restore a db.</li>
-				<li>Hit admin->your server->make a backup on your client. This will lock the db server-side while it makes a copy right in the /db directory. The .db file will be tidied up and copied to .db.backup and the subdirectories will be copied to _backup as well. When the operation is complete, you can ftp/batch-copy/whatever the backup wherever you like.</li>
+				<li>In the client, hit admin->your server->make a backup. This will lock the db server-side while it makes a copy of everything server-related to server_install_dir/db/server_backup. When the operation is complete, you can ftp/batch-copy/whatever the server_backup folder wherever you like.</li>
 			</ul>
 			<h3>OMG EVERYTHING WENT WRONG</h3>
 			<p>If you get to a point where you can no longer boot the repository, try running SQLite Studio and opening server.db. If the issue is simple—like manually changing the port number—you may be in luck. Send me an email if it is tricky.</p>

include/ClientCaches.py

@@ -12,6 +12,7 @@ import itertools
 import os
 import random
 import Queue
+import shutil
 import threading
 import time
 import urllib
@@ -190,14 +191,40 @@ class ClientFilesManager( object ):
         
         self._lock = threading.Lock()
         
-        # fetch the current exact mapping from the db
+        self._prefixes_to_locations = {}
         
         self._Reinit()
         
     
-    def _Reinit( self ):
+    def _GetLocation( self, hash ):
         
-        self._prefixes_to_locations = self._controller.Read( 'client_files_locations' )
+        hash_encoded = hash.encode( 'hex' )
+        
+        prefix = hash_encoded[:2]
+        
+        location = self._prefixes_to_locations[ prefix ]
+        
+        return location
+        
+    
+    def _GetRecoverTuple( self ):
+        
+        paths = { path for path in self._prefixes_to_locations.values() }
+        
+        for path in paths:
+            
+            for prefix in HydrusData.IterateHexPrefixes():
+                
+                correct_path = self._prefixes_to_locations[ prefix ]
+                
+                if path != correct_path and os.path.exists( os.path.join( path, prefix ) ):
+                    
+                    return ( prefix, path, correct_path )
+                    
+                
+            
+        
+        return None
         
     
     def _GetRebalanceTuple( self ):
@@ -208,13 +235,21 @@ class ClientFilesManager( object ):
         
         paths_to_normalised_ideal_weights = { path : weight / total_weight for ( path, weight ) in paths_to_ideal_weights.items() }
         
-        current_paths_to_normalised_weights = {}
+        current_paths_to_normalised_weights = collections.defaultdict( lambda: 0 )
         
         for ( prefix, path ) in self._prefixes_to_locations.items():
             
             current_paths_to_normalised_weights[ path ] += 1.0 / 256
             
         
+        for path in current_paths_to_normalised_weights.keys():
+            
+            if path not in paths_to_normalised_ideal_weights:
+                
+                paths_to_normalised_ideal_weights[ path ] = 0.0
+                
+            
+        
         #
         
         overweight_paths = []
@@ -252,7 +287,11 @@ class ClientFilesManager( object ):
             overweight_path = overweight_paths.pop( 0 )
             underweight_path = underweight_paths.pop( 0 )
             
-            for ( prefix, path ) in self._prefixes_to_locations.items():
+            prefixes_and_paths = self._prefixes_to_locations.items()
+            
+            random.shuffle( prefixes_and_paths )
+            
+            for ( prefix, path ) in prefixes_and_paths:
                 
                 if path == overweight_path:
                     
@@ -262,6 +301,79 @@ class ClientFilesManager( object ):
             
         
     
+    def _IterateAllFilePaths( self ):
+        
+        for ( prefix, location ) in self._prefixes_to_locations.items():
+            
+            dir = os.path.join( location, prefix )
+            
+            next_filenames = os.listdir( dir )
+            
+            for filename in next_filenames:
+                
+                yield os.path.join( dir, filename )
+                
+            
+        
+    
+    def _Reinit( self ):
+        
+        self._prefixes_to_locations = self._controller.Read( 'client_files_locations' )
+        
+    
+    def GetExpectedFilePath( self, hash, mime ):
+        
+        with self._lock:
+            
+            location = self._GetLocation( hash )
+            
+            return ClientFiles.GetExpectedFilePath( location, hash, mime )
+            
+        
+    
+    def GetFilePath( self, hash, mime = None ):
+        
+        with self._lock:
+            
+            location = self._GetLocation( hash )
+            
+            return ClientFiles.GetFilePath( location, hash, mime )
+            
+        
+    
+    def IterateAllFileHashes( self ):
+        
+        with self._lock:
+            
+            for path in self._IterateAllFilePaths():
+                
+                ( base, filename ) = os.path.split( path )
+                
+                result = filename.split( '.', 1 )
+                
+                if len( result ) != 2: continue
+                
+                ( hash_encoded, ext ) = result
+                
+                try: hash = hash_encoded.decode( 'hex' )
+                except TypeError: continue
+                
+                yield hash
+                
+            
+        
+    
+    def IterateAllFilePaths( self ):
+        
+        with self._lock:
+            
+            for path in self._IterateAllFilePaths():
+                
+                yield path
+                
+            
+        
+    
     def Rebalance( self, partial = True ):
         
         with self._lock:
@@ -272,7 +384,18 @@ class ClientFilesManager( object ):
                 
                 ( prefix, overweight_path, underweight_path ) = rebalance_tuple
                 
-                self._controller.Write( 'move_client_files', prefix, overweight_path, underweight_path )
+                text = 'Moving \'' + prefix + '\' files from ' + overweight_path + ' to ' + underweight_path
+                
+                if partial:
+                    
+                    HydrusData.Print( text )
+                    
+                else:
+                    
+                    HydrusData.ShowText( text )
+                    
+                
+                self._controller.Write( 'relocate_client_files', prefix, overweight_path, underweight_path )
                 
                 self._Reinit()
                 
@@ -284,6 +407,49 @@ class ClientFilesManager( object ):
                 rebalance_tuple = self._GetRebalanceTuple()
                 
             
+            recover_tuple = self._GetRecoverTuple()
+            
+            while recover_tuple is not None:
+                
+                ( prefix, incorrect_path, correct_path ) = recover_tuple
+                
+                text = 'Recovering \'' + prefix + '\' files from ' + incorrect_path + ' to ' + correct_path
+                
+                if partial:
+                    
+                    HydrusData.Print( text )
+                    
+                else:
+                    
+                    HydrusData.ShowText( text )
+                    
+                
+                full_incorrect_path = os.path.join( incorrect_path, prefix )
+                full_correct_path = os.path.join( correct_path, prefix )
+                
+                HydrusPaths.CopyAndMergeTree( full_incorrect_path, full_correct_path )
+                
+                try: HydrusPaths.RecyclePath( full_incorrect_path )
+                except:
+                    
+                    HydrusData.ShowText( 'After recovering some files, attempting to remove ' + full_incorrect_path + ' failed.' )
+                    
+                    return
+                    
+                
+                if partial:
+                    
+                    break
+                    
+                
+                recover_tuple = self._GetRecoverTuple()
+                
+            
+        
+        if not partial:
+            
+            HydrusData.ShowText( 'All folders balanced!' )
+            
         
     
 class DataCache( object ):
@@ -1392,7 +1558,10 @@ class TagSiblingsManager( object ):
                     new_predicate.AddToCount( HC.CURRENT, current_count )
                     new_predicate.AddToCount( HC.PENDING, pending_count )
                     
-                else: tags_to_include_in_results.add( tag )
+                else:
+                    
+                    tags_to_include_in_results.add( tag )
+                    
                 
             
             results.extend( [ tags_to_predicates[ tag ] for tag in tags_to_include_in_results ] )

include/ClientController.py

@@ -265,31 +265,29 @@ class Controller( HydrusController.HydrusController ):
     
     def CurrentlyIdle( self ):
         
-        # the existence of an idle test permits a True result
-        # any single fail vetoes a True
+        idle_normal = self._options[ 'idle_normal' ]
+        idle_period = self._options[ 'idle_period' ]
+        idle_mouse_period = self._options[ 'idle_mouse_period' ]
         
         possibly_idle = False
         definitely_not_idle = False
         
-        if self._options[ 'idle_period' ] > 0:
+        if idle_normal:
             
-            if HydrusData.TimeHasPassed( self._timestamps[ 'last_user_action' ] + self._options[ 'idle_period' ] ):
-                
-                possibly_idle = True
-                
-            else:
+            possibly_idle = True
+            
+        
+        if idle_period is not None:
+            
+            if not HydrusData.TimeHasPassed( self._timestamps[ 'last_user_action' ] + idle_period ):
                 
                 definitely_not_idle = True
                 
             
         
-        if self._options[ 'idle_mouse_period' ] > 0:
+        if idle_mouse_period is not None:
             
-            if HydrusData.TimeHasPassed( self._timestamps[ 'last_mouse_action' ] + self._options[ 'idle_mouse_period' ] ):
-                
-                possibly_idle = True
-                
-            else:
+            if not HydrusData.TimeHasPassed( self._timestamps[ 'last_mouse_action' ] + idle_mouse_period ):
                 
                 definitely_not_idle = True
                 
@@ -386,6 +384,11 @@ class Controller( HydrusController.HydrusController ):
         self.pub( 'refresh_status' )
         
     
+    def GetClientFilesManager( self ):
+        
+        return self._client_files_manager
+        
+    
     def GetDB( self ): return self._db
     
     def GetGUI( self ): return self._gui
@@ -420,6 +423,8 @@ class Controller( HydrusController.HydrusController ):
         
         self._services_manager = ClientCaches.ServicesManager( self )
         
+        self._client_files_manager = ClientCaches.ClientFilesManager( self )
+        
         self._managers[ 'hydrus_sessions' ] = ClientCaches.HydrusSessionManager( self )
         self._managers[ 'local_booru' ] = ClientCaches.LocalBooruCache( self )
         self._managers[ 'tag_censorship' ] = ClientCaches.TagCensorshipManager( self )
@@ -503,6 +508,7 @@ class Controller( HydrusController.HydrusController ):
         self._daemons.append( HydrusThreading.DAEMONWorker( self, 'CheckExportFolders', ClientDaemons.DAEMONCheckExportFolders, ( 'notify_restart_export_folders_daemon', 'notify_new_export_folders' ), period = 180 ) )
         self._daemons.append( HydrusThreading.DAEMONWorker( self, 'DownloadFiles', ClientDaemons.DAEMONDownloadFiles, ( 'notify_new_downloads', 'notify_new_permissions' ), pre_callable_wait = 0 ) )
         self._daemons.append( HydrusThreading.DAEMONWorker( self, 'MaintainTrash', ClientDaemons.DAEMONMaintainTrash, init_wait = 60 ) )
+        self._daemons.append( HydrusThreading.DAEMONWorker( self, 'RebalanceClientFiles', ClientDaemons.DAEMONRebalanceClientFiles, period = 3600 ) )
         self._daemons.append( HydrusThreading.DAEMONWorker( self, 'SynchroniseAccounts', ClientDaemons.DAEMONSynchroniseAccounts, ( 'permissions_are_stale', ) ) )
         self._daemons.append( HydrusThreading.DAEMONWorker( self, 'SynchroniseRepositories', ClientDaemons.DAEMONSynchroniseRepositories, ( 'notify_restart_repo_sync_daemon', 'notify_new_permissions' ), period = 360 ) )
         self._daemons.append( HydrusThreading.DAEMONWorker( self, 'SynchroniseSubscriptions', ClientDaemons.DAEMONSynchroniseSubscriptions, ( 'notify_restart_subs_sync_daemon', 'notify_new_subscriptions' ), period = 360, init_wait = 120 ) )
@@ -520,9 +526,14 @@ class Controller( HydrusController.HydrusController ):
         
         shutdown_timestamps = self.Read( 'shutdown_timestamps' )
         
-        if self._options[ 'maintenance_vacuum_period' ] != 0:
+        maintenance_vacuum_period = self._options[ 'maintenance_vacuum_period' ]
+        
+        if maintenance_vacuum_period is not None and maintenance_vacuum_period > 0:
             
-            if now - shutdown_timestamps[ CC.SHUTDOWN_TIMESTAMP_VACUUM ] > self._options[ 'maintenance_vacuum_period' ]: self.WriteSynchronous( 'vacuum' )
+            if HydrusData.TimeHasPassed( shutdown_timestamps[ CC.SHUTDOWN_TIMESTAMP_VACUUM ] + maintenance_vacuum_period ):
+                
+                self.WriteSynchronous( 'vacuum' )
+                
             
         
         if self._timestamps[ 'last_service_info_cache_fatten' ] == 0:
@@ -530,7 +541,7 @@ class Controller( HydrusController.HydrusController ):
             self._timestamps[ 'last_service_info_cache_fatten' ] = HydrusData.GetNow()
             
         
-        if now - self._timestamps[ 'last_service_info_cache_fatten' ] > 60 * 20:
+        if HydrusData.TimeHasPassed( self._timestamps[ 'last_service_info_cache_fatten' ] + ( 60 * 20 ) ):
             
             self.pub( 'splash_set_status_text', 'fattening service info' )
             
@@ -868,24 +879,31 @@ class Controller( HydrusController.HydrusController ):
         
     
     def SystemBusy( self ):
+    
+        max_cpu = self._options[ 'idle_cpu_max' ]
         
-        if HydrusData.TimeHasPassed( self._timestamps[ 'last_cpu_check' ] + 60 ):
+        if max_cpu is None:
             
-            max_cpu = self._options[ 'idle_cpu_max' ]
+            self._system_busy = False
             
-            cpu_times = psutil.cpu_percent( percpu = True )
+        else:
             
-            if True in ( cpu_time > max_cpu for cpu_time in cpu_times ):
+            if HydrusData.TimeHasPassed( self._timestamps[ 'last_cpu_check' ] + 60 ):
                 
-                self._system_busy = True
+                cpu_times = psutil.cpu_percent( percpu = True )
                 
-            else:
+                if True in ( cpu_time > max_cpu for cpu_time in cpu_times ):
+                    
+                    self._system_busy = True
+                    
+                else:
+                    
+                    self._system_busy = False
+                    
                 
-                self._system_busy = False
+                self._timestamps[ 'last_cpu_check' ] = HydrusData.GetNow()
                 
             
-            self._timestamps[ 'last_cpu_check' ] = HydrusData.GetNow()
-            
         
         return self._system_busy
         
@@ -896,9 +914,11 @@ class Controller( HydrusController.HydrusController ):
         
         shutdown_timestamps = self.Read( 'shutdown_timestamps' )
         
-        if self._options[ 'maintenance_vacuum_period' ] != 0:
+        maintenance_vacuum_period = self._options[ 'maintenance_vacuum_period' ]
+        
+        if maintenance_vacuum_period is not None and maintenance_vacuum_period > 0:
             
-            if now - shutdown_timestamps[ CC.SHUTDOWN_TIMESTAMP_VACUUM ] > self._options[ 'maintenance_vacuum_period' ]:
+            if HydrusData.TimeHasPassed( shutdown_timestamps[ CC.SHUTDOWN_TIMESTAMP_VACUUM ] + maintenance_vacuum_period ):
                 
                 return True
                 

include/ClientDB.py

@@ -832,9 +832,11 @@ class MessageDB( object ):
         
         files = []
         
+        client_files_manager = self._controller.GetClientFilesManager()        
+        
         for hash in attachment_hashes:
             
-            path = ClientFiles.GetFilePath( hash )
+            path = client_files_manager.GetFilePath( hash )
             
             with open( path, 'rb' ) as f: file = f.read()
             
@@ -876,9 +878,11 @@ class MessageDB( object ):
         
         files = []
         
+        client_files_manager = self._controller.GetClientFilesManager()
+        
         for hash in attachment_hashes:
             
-            path = ClientFiles.GetFilePath( hash )
+            path = client_files_manager.GetFilePath( hash )
             
             with open( path, 'rb' ) as f: file = f.read()
             
@@ -1203,8 +1207,8 @@ class DB( HydrusDB.HydrusDB ):
             ClientData.DeletePath( deletee_path )
             
         
-        shutil.copy( self._db_path, os.path.join( path, 'client.db' ) )
-        if os.path.exists( self._db_path + '-wal' ): shutil.copy( self._db_path + '-wal', os.path.join( path, 'client.db-wal' ) )
+        shutil.copy2( self._db_path, os.path.join( path, 'client.db' ) )
+        if os.path.exists( self._db_path + '-wal' ): shutil.copy2( self._db_path + '-wal', os.path.join( path, 'client.db-wal' ) )
         
         shutil.copytree( HC.CLIENT_ARCHIVES_DIR, os.path.join( path, 'client_archives' ) ) 
         shutil.copytree( HC.CLIENT_FILES_DIR, os.path.join( path, 'client_files' ) )
@@ -1279,6 +1283,8 @@ class DB( HydrusDB.HydrusDB ):
         missing_count = 0
         deletee_hash_ids = []
         
+        client_files_manager = self._controller.GetClientFilesManager()
+        
         for ( i, ( hash_id, mime ) ) in enumerate( info ):
             
             ( i_paused, should_quit ) = job_key.WaitIfNeeded()
@@ -1293,7 +1299,7 @@ class DB( HydrusDB.HydrusDB ):
             
             hash = self._GetHash( hash_id )
             
-            try: path = ClientFiles.GetFilePath( hash, mime )
+            try: path = client_files_manager.GetFilePath( hash, mime )
             except HydrusExceptions.NotFoundException:
                 
                 deletee_hash_ids.append( hash_id )
@@ -1374,6 +1380,8 @@ class DB( HydrusDB.HydrusDB ):
     
     def _CopyFiles( self, hashes ):
         
+        client_files_manager = self._controller.GetClientFilesManager()
+        
         if len( hashes ) > 0:
             
             error_messages = set()
@@ -1384,7 +1392,7 @@ class DB( HydrusDB.HydrusDB ):
                 
                 try:
                     
-                    path = ClientFiles.GetFilePath( hash )
+                    path = client_files_manager.GetFilePath( hash )
                     
                     paths.append( path )
                     
@@ -1409,15 +1417,13 @@ class DB( HydrusDB.HydrusDB ):
         if not os.path.exists( HC.CLIENT_THUMBNAILS_DIR ): os.mkdir( HC.CLIENT_THUMBNAILS_DIR )
         if not os.path.exists( HC.CLIENT_UPDATES_DIR ): os.mkdir( HC.CLIENT_UPDATES_DIR )
         
-        hex_chars = '0123456789abcdef'
-        
-        for ( one, two ) in itertools.product( hex_chars, hex_chars ):
+        for prefix in HydrusData.IterateHexPrefixes():
             
-            dir = os.path.join( HC.CLIENT_FILES_DIR, one + two )
+            dir = os.path.join( HC.CLIENT_FILES_DIR, prefix )
             
             if not os.path.exists( dir ): os.mkdir( dir )
             
-            dir = os.path.join( HC.CLIENT_THUMBNAILS_DIR, one + two )
+            dir = os.path.join( HC.CLIENT_THUMBNAILS_DIR, prefix )
             
             if not os.path.exists( dir ): os.mkdir( dir )
             
@@ -1439,6 +1445,8 @@ class DB( HydrusDB.HydrusDB ):
         self._c.execute( 'CREATE TABLE autocomplete_tags_cache ( file_service_id INTEGER REFERENCES services ( service_id ) ON DELETE CASCADE, tag_service_id INTEGER REFERENCES services ( service_id ) ON DELETE CASCADE, namespace_id INTEGER, tag_id INTEGER, current_count INTEGER, pending_count INTEGER, PRIMARY KEY ( file_service_id, tag_service_id, namespace_id, tag_id ) );' )
         self._c.execute( 'CREATE INDEX autocomplete_tags_cache_tag_service_id_namespace_id_tag_id_index ON autocomplete_tags_cache ( tag_service_id, namespace_id, tag_id );' )
         
+        self._c.execute( 'CREATE TABLE client_files_locations ( prefix TEXT, location TEXT );' )
+        
         self._c.execute( 'CREATE TABLE contacts ( contact_id INTEGER PRIMARY KEY, contact_key BLOB_BYTES, public_key TEXT, name TEXT, host TEXT, port INTEGER );' )
         self._c.execute( 'CREATE UNIQUE INDEX contacts_contact_key_index ON contacts ( contact_key );' )
         self._c.execute( 'CREATE UNIQUE INDEX contacts_name_index ON contacts ( name );' )
@@ -1572,6 +1580,13 @@ class DB( HydrusDB.HydrusDB ):
         
         # inserts
         
+        location = HydrusPaths.ConvertAbsPathToPortablePath( HC.CLIENT_FILES_DIR )
+        
+        for prefix in HydrusData.IterateHexPrefixes():
+            
+            self._c.execute( 'INSERT INTO client_files_locations ( prefix, location ) VALUES ( ?, ? );', ( prefix, location ) )
+            
+        
         init_service_info = []
         
         init_service_info.append( ( CC.LOCAL_FILE_SERVICE_KEY, HC.LOCAL_FILE, CC.LOCAL_FILE_SERVICE_KEY ) )
@@ -1716,7 +1731,9 @@ class DB( HydrusDB.HydrusDB ):
         
         time.sleep( 1 )
         
-        for hash in ClientFiles.IterateAllFileHashes():
+        client_files_manager = self._controller.GetClientFilesManager()
+        
+        for hash in client_files_manager.IterateAllFileHashes():
             
             ( i_paused, should_quit ) = job_key.WaitIfNeeded()
             
@@ -1727,7 +1744,7 @@ class DB( HydrusDB.HydrusDB ):
             
             if hash in deletee_hashes:
                 
-                try: path = ClientFiles.GetFilePath( hash )
+                try: path = client_files_manager.GetFilePath( hash )
                 except HydrusExceptions.NotFoundException: continue
                 
                 try:
@@ -1866,9 +1883,11 @@ class DB( HydrusDB.HydrusDB ):
             
             file_hashes = self._GetHashes( deletable_file_hash_ids )
             
+            client_files_manager = self._controller.GetClientFilesManager()
+            
             for hash in file_hashes:
                 
-                try: path = ClientFiles.GetFilePath( hash )
+                try: path = client_files_manager.GetFilePath( hash )
                 except HydrusExceptions.NotFoundException: continue
                 
                 deletee_paths.add( path )
@@ -2268,6 +2287,13 @@ class DB( HydrusDB.HydrusDB ):
         return predicates
         
     
+    def _GetClientFilesLocations( self ):
+        
+        result = { prefix : HydrusPaths.ConvertPortablePathToAbsPath( location ) for ( prefix, location ) in self._c.execute( 'SELECT prefix, location FROM client_files_locations;' ) }
+        
+        return result
+        
+    
     def _GetDownloads( self ): return { hash for ( hash, ) in self._c.execute( 'SELECT hash FROM file_transfers, hashes USING ( hash_id ) WHERE service_id = ?;', ( self._local_file_service_id, ) ) }
     
     def _GetFileHash( self, hash, hash_type ):
@@ -3578,6 +3604,27 @@ class DB( HydrusDB.HydrusDB ):
             self._service_cache[ service_id ] = service
             
         
+        if service.GetServiceType() == HC.LOCAL_BOORU:
+            
+            info = service.GetInfo()
+            
+            current_time_struct = time.gmtime()
+            
+            ( current_year, current_month ) = ( current_time_struct.tm_year, current_time_struct.tm_mon )
+            
+            ( booru_year, booru_month ) = info[ 'current_data_month' ]
+            
+            if current_year != booru_year or current_month != booru_month:
+                
+                info[ 'used_monthly_data' ] = 0
+                info[ 'used_monthly_requests' ] = 0
+                
+                info[ 'current_data_month' ] = ( current_year, current_month )
+                
+                self._c.execute( 'UPDATE services SET info = ? WHERE service_id = ?;', ( info, service_id ) )
+                
+            
+        
         return service
         
     
@@ -4048,11 +4095,13 @@ class DB( HydrusDB.HydrusDB ):
             
             mime = HydrusFileHandling.GetMime( path )
             
-            dest_path = ClientFiles.GetExpectedFilePath( hash, mime )
+            client_files_manager = self._controller.GetClientFilesManager()
+            
+            dest_path = client_files_manager.GetExpectedFilePath( hash, mime )
             
             if not os.path.exists( dest_path ):
                 
-                shutil.copy( path, dest_path )
+                shutil.copy2( path, dest_path )
                 
                 try: os.chmod( dest_path, stat.S_IWRITE | stat.S_IREAD )
                 except: pass
@@ -4265,7 +4314,7 @@ class DB( HydrusDB.HydrusDB ):
                 
                 content_update_index_string = 'content row ' + HydrusData.ConvertValueRangeToPrettyString( c_u_p_total_weight_processed, c_u_p_num_rows ) + ': '
                 
-                HydrusGlobals.client_controller.pub( 'splash_set_status_text', content_update_index_string + 'committing' + update_speed_string )
+                self._controller.pub( 'splash_set_status_text', content_update_index_string + 'committing' + update_speed_string )
                 job_key.SetVariable( 'popup_text_2', content_update_index_string + 'committing' + update_speed_string )
                 
                 job_key.SetVariable( 'popup_gauge_2', ( c_u_p_total_weight_processed, c_u_p_num_rows ) )
@@ -4900,20 +4949,6 @@ class DB( HydrusDB.HydrusDB ):
             
             info = service.GetInfo()
             
-            current_time_struct = time.gmtime()
-            
-            ( current_year, current_month ) = ( current_time_struct.tm_year, current_time_struct.tm_mon )
-            
-            ( booru_year, booru_month ) = info[ 'current_data_month' ]
-            
-            if current_year != booru_year or current_month != booru_month:
-                
-                info[ 'used_monthly_data' ] = 0
-                info[ 'used_monthly_requests' ] = 0
-                
-                info[ 'current_data_month' ] = ( current_year, current_month )
-                
-            
             info[ 'used_monthly_data' ] += sum( local_booru_requests_made )
             info[ 'used_monthly_requests' ] += len( local_booru_requests_made )
             
@@ -4929,6 +4964,7 @@ class DB( HydrusDB.HydrusDB ):
         elif action == 'tag_archive_info': result = self._GetTagArchiveInfo( *args, **kwargs )
         elif action == 'tag_archive_tags': result = self._GetTagArchiveTags( *args, **kwargs )
         elif action == 'autocomplete_predicates': result = self._GetAutocompletePredicates( *args, **kwargs )
+        elif action == 'client_files_locations': result = self._GetClientFilesLocations( *args, **kwargs )
         elif action == 'downloads': result = self._GetDownloads( *args, **kwargs )
         elif action == 'file_hash': result = self._GetFileHash( *args, **kwargs )
         elif action == 'file_query_ids': result = self._GetHashIdsFromQuery( *args, **kwargs )
@@ -4967,6 +5003,31 @@ class DB( HydrusDB.HydrusDB ):
         return result
         
     
+    def _RelocateClientFiles( self, prefix, source, dest ):
+        
+        full_source = os.path.join( source, prefix )
+        full_dest = os.path.join( dest, prefix )
+        
+        if os.path.exists( full_source ):
+            
+            HydrusPaths.CopyAndMergeTree( full_source, full_dest )
+            
+        elif not os.path.exists( full_dest ):
+            
+            os.mkdir( full_dest )
+            
+        
+        portable_dest = HydrusPaths.ConvertAbsPathToPortablePath( dest )
+        
+        self._c.execute( 'UPDATE client_files_locations SET location = ? WHERE prefix = ?;', ( portable_dest, prefix ) )
+        
+        if os.path.exists( full_source ):
+            
+            try: HydrusPaths.RecyclePath( full_source )
+            except: pass
+            
+        
+    
     def _ResetService( self, service_key, delete_updates = False ):
         
         service_id = self._GetServiceId( service_key )
@@ -5273,88 +5334,6 @@ class DB( HydrusDB.HydrusDB ):
         
         self._controller.pub( 'splash_set_title_text', 'updating db to v' + str( version + 1 ) )
         
-        if version == 131:
-            
-            service_info = self._c.execute( 'SELECT service_id, info FROM services;' ).fetchall()
-            
-            for ( service_id, info ) in service_info:
-                
-                if 'account' in info:
-                    
-                    info[ 'account' ] = HydrusData.GetUnknownAccount()
-                    
-                    self._c.execute( 'UPDATE services SET info = ? WHERE service_id = ?;', ( info, service_id ) )
-                    
-                
-            
-        
-        if version == 132:
-            
-            self._c.execute( 'DELETE FROM service_info WHERE info_type = ?;', ( HC.SERVICE_INFO_NUM_FILES, ) )
-            
-            #
-            
-            options = self._GetOptions()
-            
-            client_size = options[ 'client_size' ]
-            
-            client_size[ 'fs_fullscreen' ] = True
-            
-            client_size[ 'gui_fullscreen' ] = False
-            
-            del options[ 'fullscreen_borderless' ]
-            
-            self._c.execute( 'UPDATE options SET options = ?;', ( options, ) )
-            
-        
-        if version == 135:
-            
-            if not os.path.exists( HC.CLIENT_ARCHIVES_DIR ): os.mkdir( HC.CLIENT_ARCHIVES_DIR )
-            
-            #
-            
-            extra_hashes_data = self._c.execute( 'SELECT * FROM local_hashes;' ).fetchall()
-            
-            self._c.execute( 'DROP TABLE local_hashes;' )
-            
-            self._c.execute( 'CREATE TABLE local_hashes ( hash_id INTEGER PRIMARY KEY, md5 BLOB_BYTES, sha1 BLOB_BYTES, sha512 BLOB_BYTES );' )
-            self._c.execute( 'CREATE INDEX local_hashes_md5_index ON local_hashes ( md5 );' )
-            self._c.execute( 'CREATE INDEX local_hashes_sha1_index ON local_hashes ( sha1 );' )
-            self._c.execute( 'CREATE INDEX local_hashes_sha512_index ON local_hashes ( sha512 );' )
-            
-            for ( i, ( hash_id, md5, sha1 ) ) in enumerate( extra_hashes_data ):
-                
-                hash = self._GetHash( hash_id )
-                
-                try: path = ClientFiles.GetFilePath( hash )
-                except HydrusExceptions.NotFoundException: continue
-                
-                h_sha512 = hashlib.sha512()
-                
-                with open( path, 'rb' ) as f:
-                    
-                    for block in HydrusPaths.ReadFileLikeAsBlocks( f ): h_sha512.update( block )
-                    
-                    sha512 = h_sha512.digest()
-                    
-                
-                self._c.execute( 'INSERT INTO local_hashes ( hash_id, md5, sha1, sha512 ) VALUES ( ?, ?, ?, ? );', ( hash_id, sqlite3.Binary( md5 ), sqlite3.Binary( sha1 ), sqlite3.Binary( sha512 ) ) )
-                
-                if i % 100 == 0: self._controller.pub( 'splash_set_status_text', 'generating sha512 hashes: ' + HydrusData.ConvertIntToPrettyString( i ) )
-                
-            
-            #
-            
-            tag_service_info = self._c.execute( 'SELECT service_id, info FROM services WHERE service_type IN ' + HydrusData.SplayListForDB( HC.TAG_SERVICES ) + ';' ).fetchall()
-            
-            for ( service_id, info ) in tag_service_info:
-                
-                info[ 'tag_archive_sync' ] = {}
-                
-                self._c.execute( 'UPDATE services SET info = ? WHERE service_id = ?;', ( info, service_id ) )
-                
-            
-        
         if version == 136:
             
             result = self._c.execute( 'SELECT tag_id FROM tags WHERE tag = ?;', ( '', ) ).fetchone()
@@ -5733,7 +5712,22 @@ class DB( HydrusDB.HydrusDB ):
             
             #
             
-            for ( i, path ) in enumerate( ClientFiles.IterateAllFilePaths() ):
+            def iterate_all_file_paths():
+                
+                for prefix in HydrusData.IterateHexPrefixes():
+                    
+                    dir = os.path.join( HC.CLIENT_FILES_DIR, prefix )
+                    
+                    next_paths = os.listdir( dir )
+                    
+                    for path in next_paths:
+                        
+                        yield os.path.join( dir, path )
+                        
+                    
+                
+            
+            for ( i, path ) in enumerate( iterate_all_file_paths() ):
                 
                 try: os.chmod( path, stat.S_IWRITE | stat.S_IREAD )
                 except: pass
@@ -5862,7 +5856,7 @@ class DB( HydrusDB.HydrusDB ):
         if version == 175:
             
             HC.options = self._GetOptions()
-            HydrusGlobals.client_controller._options = HC.options
+            self._controller._options = HC.options
             
             self._c.execute( 'DELETE FROM yaml_dumps WHERE dump_type = ?;', ( YAML_DUMP_ID_IMPORT_FOLDER, ) )
             
@@ -6160,18 +6154,14 @@ class DB( HydrusDB.HydrusDB ):
                 
                 hash = self._GetHash( hash_id )
                 
-                try:
-                    
-                    file_path = ClientFiles.GetFilePath( hash, HC.APPLICATION_FLASH )
-                    
-                except HydrusExceptions.NotFoundException:
-                    
-                    continue
-                    
+                file_path = ClientFiles.GetExpectedFilePath( HC.CLIENT_FILES_DIR, hash, HC.APPLICATION_FLASH )
                 
-                thumbnail = HydrusFileHandling.GenerateThumbnail( file_path )
-                
-                self._AddThumbnails( [ ( hash, thumbnail ) ] )
+                if os.path.exists( file_path ):
+                    
+                    thumbnail = HydrusFileHandling.GenerateThumbnail( file_path )
+                    
+                    self._AddThumbnails( [ ( hash, thumbnail ) ] )
+                    
                 
             
             #
@@ -6179,6 +6169,18 @@ class DB( HydrusDB.HydrusDB ):
             self._c.execute( 'DELETE FROM service_info WHERE info_type IN ( ?, ? );', ( HC.SERVICE_INFO_NUM_THUMBNAILS, HC.SERVICE_INFO_NUM_THUMBNAILS_LOCAL ) )
             
         
+        if version == 183:
+            
+            self._c.execute( 'CREATE TABLE client_files_locations ( prefix TEXT, location TEXT );' )
+            
+            location = HydrusPaths.ConvertAbsPathToPortablePath( HC.CLIENT_FILES_DIR )
+            
+            for prefix in HydrusData.IterateHexPrefixes():
+                
+                self._c.execute( 'INSERT INTO client_files_locations ( prefix, location ) VALUES ( ?, ? );', ( prefix, location ) )
+                
+            
+        
         self._controller.pub( 'splash_set_title_text', 'updating db to v' + str( version + 1 ) )
         
         self._c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
@@ -6693,7 +6695,7 @@ class DB( HydrusDB.HydrusDB ):
             
         except sqlite3.OperationalError:
             
-            HC.options[ 'maintenance_vacuum_period' ] = 0
+            HC.options[ 'maintenance_vacuum_period' ] = None
             
             self._SaveOptions( HC.options )
             
@@ -6749,6 +6751,7 @@ class DB( HydrusDB.HydrusDB ):
         elif action == 'import_file': result = self._ImportFile( *args, **kwargs )
         elif action == 'local_booru_share': result = self._SetYAMLDump( YAML_DUMP_ID_LOCAL_BOORU, *args, **kwargs )
         elif action == 'pixiv_account': result = self._SetYAMLDump( YAML_DUMP_ID_SINGLE, 'pixiv_account', *args, **kwargs )
+        elif action == 'relocate_client_files': result = self._RelocateClientFiles( *args, **kwargs )
         elif action == 'remote_booru': result = self._SetYAMLDump( YAML_DUMP_ID_REMOTE_BOORU, *args, **kwargs )
         elif action == 'reset_service': result = self._ResetService( *args, **kwargs )
         elif action == 'save_options': result = self._SaveOptions( *args, **kwargs )
@@ -6792,10 +6795,10 @@ class DB( HydrusDB.HydrusDB ):
                 
             
         
-        shutil.copy( os.path.join( path, 'client.db' ), self._db_path )
+        shutil.copy2( os.path.join( path, 'client.db' ), self._db_path )
         
         wal_path = os.path.join( path, 'client.db-wal' )
-        if os.path.exists( wal_path ): shutil.copy( wal_path, self._db_path + '-wal' )
+        if os.path.exists( wal_path ): shutil.copy2( wal_path, self._db_path + '-wal' )
         
         shutil.copytree( os.path.join( path, 'client_archives' ), HC.CLIENT_ARCHIVES_DIR )
         shutil.copytree( os.path.join( path, 'client_files' ), HC.CLIENT_FILES_DIR )

include/ClientDaemons.py

@@ -234,6 +234,13 @@ def DAEMONMaintainTrash( controller ):
             
         
     
+def DAEMONRebalanceClientFiles( controller ):
+    
+    if controller.CurrentlyIdle() and not controller.SystemBusy():
+        
+        controller.GetClientFilesManager().Rebalance()
+        
+    
 def DAEMONSynchroniseAccounts( controller ):
     
     services = controller.GetServicesManager().GetServices( HC.RESTRICTED_SERVICES )

include/ClientData.py

@@ -292,6 +292,20 @@ def ShowTextClient( text ):
     
     HydrusGlobals.client_controller.pub( 'message', job_key )
     
+def WaitPolitely( page_key = None ):
+    
+    if page_key is not None:
+        
+        HydrusGlobals.client_controller.pub( 'waiting_politely', page_key, True )
+        
+    
+    time.sleep( HC.options[ 'website_download_polite_wait' ] )
+    
+    if page_key is not None:
+        
+        HydrusGlobals.client_controller.pub( 'waiting_politely', page_key, False )
+        
+    
 class Booru( HydrusData.HydrusYAMLBase ):
     
     yaml_tag = u'!Booru'
@@ -480,6 +494,13 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
             
         
     
+    def SetClientFilesLocationsToIdealWeights( self, locations_to_weights ):
+        
+        portable_locations_and_weights = [ ( HydrusPaths.ConvertAbsPathToPortablePath( location ), float( weight ) ) for ( location, weight ) in locations_to_weights.items() ]
+        
+        self._dictionary[ 'client_files_locations_ideal_weights' ] = portable_locations_and_weights
+        
+    
     def SetDefaultImportTagOptions( self, gallery_identifier, import_tag_options ):
         
         with self._lock:
@@ -2169,6 +2190,11 @@ class Service( HydrusData.HydrusYAMLBase ):
                     
                     service_update_package = HydrusSerialisable.CreateFromString( obj_string )
                     
+                    if not isinstance( service_update_package, HydrusData.ServerToClientServiceUpdatePackage ):
+                        
+                        raise Exception()
+                        
+                    
                 except:
                     
                     self._ReportSyncProcessingError( path, 'did not parse' )
@@ -2230,6 +2256,11 @@ class Service( HydrusData.HydrusYAMLBase ):
                         
                         content_update_package = HydrusSerialisable.CreateFromString( obj_string )
                         
+                        if not isinstance( content_update_package, HydrusData.ServerToClientContentUpdatePackage ):
+                            
+                            raise Exception()
+                            
+                        
                     except:
                         
                         self._ReportSyncProcessingError( path, 'did not parse' )

include/ClientDefaults.py

@@ -24,11 +24,13 @@ def GetClientDefaultOptions():
     options[ 'num_autocomplete_chars' ] = 2
     options[ 'gui_capitalisation' ] = False
     options[ 'default_gui_session' ] = 'just a blank page'
+    options[ 'fetch_ac_results_automatically' ] = True
     options[ 'ac_timings' ] = ( 3, 500, 250 )
     options[ 'thread_checker_timings' ] = ( 3, 1200 )
     options[ 'idle_period' ] = 60 * 30
     options[ 'idle_mouse_period' ] = 60 * 10
     options[ 'idle_cpu_max' ] = 50
+    options[ 'idle_normal' ] = True
     options[ 'idle_shutdown' ] = CC.IDLE_ON_SHUTDOWN_ASK_FIRST
     options[ 'idle_shutdown_max_minutes' ] = 30
     options[ 'maintenance_delete_orphans_period' ] = 86400 * 3

include/ClientFiles.py

@@ -105,7 +105,7 @@ def GetAllPaths( raw_paths ):
     gc.collect()
     
     return file_paths
-
+    
 def GetAllThumbnailHashes():
     
     thumbnail_hashes = set()
@@ -116,14 +116,6 @@ def GetAllThumbnailHashes():
         
     
     return thumbnail_hashes
-
-def GetExpectedFilePath( hash, mime ):
-    
-    hash_encoded = hash.encode( 'hex' )
-    
-    first_two_chars = hash_encoded[:2]
-    
-    return os.path.join( HC.CLIENT_FILES_DIR, first_two_chars, hash_encoded + HC.mime_ext_lookup[ mime ] )
     
 def GetExportPath():
     
@@ -147,35 +139,13 @@ def GetExportPath():
     
     return path
     
-def GetFilePath( hash, mime = None ):
+def GetExpectedFilePath( location, hash, mime ):
     
-    if mime is None:
-        
-        path = None
-        
-        for potential_mime in HC.ALLOWED_MIMES:
-            
-            potential_path = GetExpectedFilePath( hash, potential_mime )
-            
-            if os.path.exists( potential_path ):
-                
-                path = potential_path
-                
-                break
-                
-            
-        
-    else:
-        
-        path = GetExpectedFilePath( hash, mime )
-        
+    hash_encoded = hash.encode( 'hex' )
     
-    if path is None or not os.path.exists( path ):
-        
-        raise HydrusExceptions.NotFoundException( 'File not found!' )
-        
+    prefix = hash_encoded[:2]
     
-    return path
+    return os.path.join( location, prefix, hash_encoded + HC.mime_ext_lookup[ mime ] )
     
 def GetExpectedThumbnailPath( hash, full_size = True ):
     
@@ -191,7 +161,37 @@ def GetExpectedThumbnailPath( hash, full_size = True ):
         
     
     return path
-
+    
+def GetFilePath( location, hash, mime = None ):
+    
+    if mime is None:
+        
+        path = None
+        
+        for potential_mime in HC.ALLOWED_MIMES:
+            
+            potential_path = GetExpectedFilePath( location, hash, potential_mime )
+            
+            if os.path.exists( potential_path ):
+                
+                path = potential_path
+                
+                break
+                
+            
+        
+    else:
+        
+        path = GetExpectedFilePath( location, hash, mime )
+        
+    
+    if path is None or not os.path.exists( path ):
+        
+        raise HydrusExceptions.NotFoundException( 'File not found!' )
+        
+    
+    return path
+    
 def GetThumbnailPath( hash, full_size = True ):
     
     path = GetExpectedThumbnailPath( hash, full_size )
@@ -240,40 +240,6 @@ def GetExpectedUpdateDir( service_key ):
     
     return os.path.join( HC.CLIENT_UPDATES_DIR, service_key.encode( 'hex' ) )
     
-def IterateAllFileHashes():
-    
-    for path in IterateAllFilePaths():
-        
-        ( base, filename ) = os.path.split( path )
-        
-        result = filename.split( '.', 1 )
-        
-        if len( result ) != 2: continue
-        
-        ( hash_encoded, ext ) = result
-        
-        try: hash = hash_encoded.decode( 'hex' )
-        except TypeError: continue
-        
-        yield hash
-        
-    
-def IterateAllFilePaths():
-    
-    hex_chars = '0123456789abcdef'
-    
-    for ( one, two ) in itertools.product( hex_chars, hex_chars ):
-        
-        dir = os.path.join( HC.CLIENT_FILES_DIR, one + two )
-        
-        next_paths = os.listdir( dir )
-        
-        for path in next_paths:
-            
-            yield os.path.join( dir, path )
-            
-        
-
 def IterateAllThumbnailHashes():
     
     for path in IterateAllThumbnailPaths():
@@ -291,11 +257,9 @@ def IterateAllThumbnailHashes():
     
 def IterateAllThumbnailPaths():
     
-    hex_chars = '0123456789abcdef'
-    
-    for ( one, two ) in itertools.product( hex_chars, hex_chars ):
+    for prefix in HydrusData.IterateHexPrefixes():
         
-        dir = os.path.join( HC.CLIENT_THUMBNAILS_DIR, one + two )
+        dir = os.path.join( HC.CLIENT_THUMBNAILS_DIR, prefix )
         
         next_paths = os.listdir( dir )
         
@@ -413,22 +377,14 @@ class ExportFolder( HydrusSerialisable.SerialisableBaseNamed ):
     
     def DoWork( self ):
         
-        if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' checking' )
-        
         if HydrusData.TimeHasPassed( self._last_checked + self._period ):
             
-            if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' time to begin' )
-            
             folder_path = self._name
             
             if os.path.exists( folder_path ) and os.path.isdir( folder_path ):
                 
-                if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' folder checks out ok' )
-                
                 query_hash_ids = HydrusGlobals.client_controller.Read( 'file_query_ids', self._file_search_context )
                 
-                if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' results found: ' + str( len( query_hash_ids ) ) )
-                
                 query_hash_ids = list( query_hash_ids )
                 
                 random.shuffle( query_hash_ids )
@@ -445,8 +401,6 @@ class ExportFolder( HydrusSerialisable.SerialisableBaseNamed ):
                 
                 while i < len( query_hash_ids ):
                     
-                    if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' building results: ' + str( i ) + '/' + str( len( query_hash_ids ) ) )
-                    
                     if HC.options[ 'pause_export_folders_sync' ]: return
                     
                     if i == 0: ( last_i, i ) = ( 0, base )
@@ -459,83 +413,74 @@ class ExportFolder( HydrusSerialisable.SerialisableBaseNamed ):
                     media_results.extend( more_media_results )
                     
                 
-                if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' media_results: ' + str( len( media_results ) ) )
-                
                 #
                 
                 terms = ParseExportPhrase( self._phrase )
                 
                 previous_filenames = set( os.listdir( HydrusData.ToUnicode( folder_path ) ) )
                 
-                if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' existing filenames: ' + str( len( previous_filenames ) ) )
-                if HydrusGlobals.special_debug_mode:
-                    for previous_filename in previous_filenames:
-                        
-                        HydrusData.Print( previous_filename )
-                        
-                    
-                
                 sync_filenames = set()
                 
+                client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+                
                 for media_result in media_results:
                     
                     hash = media_result.GetHash()
                     mime = media_result.GetMime()
                     size = media_result.GetSize()
                     
-                    source_path = GetFilePath( hash, mime )
+                    source_path = client_files_manager.GetFilePath( hash, mime )
                     
                     filename = GenerateExportFilename( media_result, terms )
                     
                     dest_path = os.path.join( folder_path, filename )
-                    if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' dest path: ' + dest_path )
+                    
                     do_copy = True
                     
                     if filename in sync_filenames:
-                        if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' it was already attempted this run' )
+                        
                         do_copy = False
                         
                     elif os.path.exists( dest_path ):
-                        if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' it exists' )
+                        
                         dest_info = os.lstat( dest_path )
                         
                         dest_size = dest_info[6]
                         
                         if dest_size == size:
-                            if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' and the file size is the same' )
+                            
                             do_copy = False
                             
                         
-                    if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' copy decision: ' + str( do_copy ) )
+                    
                     if do_copy:
-                        if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' copy started' )
-                        shutil.copy( source_path, dest_path )
-                        shutil.copystat( source_path, dest_path )
+                        
+                        shutil.copy2( source_path, dest_path )
                         
                         try: os.chmod( dest_path, stat.S_IWRITE | stat.S_IREAD )
                         except: pass
-                        if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' copy ok' )
+                        
                     
                     sync_filenames.add( filename )
                     
-                if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' media results done' )
+                
                 if self._export_type == HC.EXPORT_FOLDER_TYPE_SYNCHRONISE:
-                    if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' inside sync delete code' )
+                    
                     deletee_filenames = previous_filenames.difference( sync_filenames )
-                    if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' delete filenames: ' + str( len( deletee_filenames ) ) )
+                    
                     for deletee_filename in deletee_filenames:
                         
                         deletee_path = os.path.join( folder_path, deletee_filename )
-                        if HydrusGlobals.special_debug_mode: HydrusData.Print( deletee_path )
+                        
                         ClientData.DeletePath( deletee_path )
                         
                     
                 
             
             self._last_checked = HydrusData.GetNow()
-            if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' writing self back to db' )
+            
             HydrusGlobals.client_controller.WriteSynchronous( 'serialisable', self )
-            if HydrusGlobals.special_debug_mode: HydrusData.ShowText( self._name + ' saved ok' )
+            
         
     
     def ToTuple( self ):

include/ClientGUI.py

@@ -830,6 +830,7 @@ class FrameGUI( ClientGUICommon.FrameThatResizes ):
             submenu = wx.Menu()
             
             submenu.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'vacuum_db' ), p( '&Vacuum' ), p( 'Rebuild the Database.' ) )
+            submenu.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'rebalance_client_files' ), p( '&Rebalance File Storage' ), p( 'Move your files around your chosen storage directories until they satisfy the weights you have set in the options.' ) )
             #submenu.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'delete_orphans' ), p( '&Delete Orphan Files' ), p( 'Go through the client\'s file store, deleting any files that are no longer needed.' ) )
             submenu.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'regenerate_thumbnails' ), p( '&Regenerate All Thumbnails' ), p( 'Delete all thumbnails and regenerate from original files.' ) )
             submenu.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'file_integrity' ), p( '&Check File Integrity' ), p( 'Review and fix all local file records.' ) )
@@ -1027,15 +1028,12 @@ class FrameGUI( ClientGUICommon.FrameThatResizes ):
             
             db_profile_mode_id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'db_profile_mode' )
             pubsub_profile_mode_id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'pubsub_profile_mode' )
-            special_debug_mode_id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'special_debug_mode' )
             
             debug = wx.Menu()
             debug.AppendCheckItem( db_profile_mode_id, p( '&DB Profile Mode' ) )
             debug.Check( db_profile_mode_id, HydrusGlobals.db_profile_mode )
             debug.AppendCheckItem( pubsub_profile_mode_id, p( '&PubSub Profile Mode' ) )
             debug.Check( pubsub_profile_mode_id, HydrusGlobals.pubsub_profile_mode )
-            debug.AppendCheckItem( special_debug_mode_id, p( '&Special Debug Mode' ) )
-            debug.Check( special_debug_mode_id, HydrusGlobals.special_debug_mode )
             debug.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'force_idle' ), p( 'Force Idle Mode' ) )
             debug.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'force_unbusy' ), p( 'Force Unbusy Mode' ) )
             debug.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetPermanentId( 'debug_garbage' ), p( 'Garbage' ) )
@@ -1431,6 +1429,21 @@ class FrameGUI( ClientGUICommon.FrameThatResizes ):
             
         
     
+    def _RebalanceClientFiles( self ):
+        
+        text = 'This will move your files around your storage directories until they satisfy the weights you have set in the options. It will also recover and folders that are in the wrong place. Use this if you have recently changed your file storage locations and want to hurry any transfers you have set up, or if you are recovering a complicated backup.'
+        text += os.linesep * 2
+        text += 'The operation will lock file access and the database. Popup messages will report its progress.'
+        
+        with ClientGUIDialogs.DialogYesNo( self, text ) as dlg:
+            
+            if dlg.ShowModal() == wx.ID_YES:
+                
+                self._controller.CallToThread( self._controller.GetClientFilesManager().Rebalance, partial = False )
+                
+            
+        
+    
     def _RefreshStatusBar( self ):
         
         page = self._notebook.GetCurrentPage()
@@ -1491,11 +1504,9 @@ class FrameGUI( ClientGUICommon.FrameThatResizes ):
                     
                     if not os.path.exists( HC.CLIENT_THUMBNAILS_DIR ): os.mkdir( HC.CLIENT_THUMBNAILS_DIR )
                     
-                    hex_chars = '0123456789abcdef'
-                    
-                    for ( one, two ) in itertools.product( hex_chars, hex_chars ):
+                    for prefix in HydrusData.IterateHexPrefixes():
                         
-                        dir = os.path.join( HC.CLIENT_THUMBNAILS_DIR, one + two )
+                        dir = os.path.join( HC.CLIENT_THUMBNAILS_DIR, prefix )
                         
                         if not os.path.exists( dir ):
                             
@@ -1505,7 +1516,7 @@ class FrameGUI( ClientGUICommon.FrameThatResizes ):
                     
                     num_broken = 0
                     
-                    for ( i, path ) in enumerate( ClientFiles.IterateAllFilePaths() ):
+                    for ( i, path ) in enumerate( self._controller.GetClientFilesManager().IterateAllFilePaths() ):
                         
                         try:
                             
@@ -1812,6 +1823,8 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
                 
                 error_messages = set()
                 
+                client_files_manager = self._controller.GetClientFilesManager()
+                
                 for ( i, media_result ) in enumerate( media_results ):
                     
                     while job_key.IsPaused() or job_key.IsCancelled():
@@ -1843,7 +1856,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
                     
                     try:
                         
-                        path = ClientFiles.GetFilePath( hash, mime )
+                        path = client_files_manager.GetFilePath( hash, mime )
                         
                         with open( path, 'rb' ) as f: file = f.read()
                         
@@ -1954,6 +1967,8 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
         
         job_key.SetVariable( 'popup_text_1', prefix + 'upload done!' )
         
+        job_key.DeleteVariable( 'popup_gauge_1' )
+        
         HydrusData.Print( job_key.ToString() )
         
         job_key.Finish()
@@ -2105,7 +2120,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
                 
             elif command == '8chan_board': webbrowser.open( 'https://8ch.net/hydrus/index.html' )
             elif command == 'file_integrity': self._CheckFileIntegrity()
-            elif command == 'help': webbrowser.open( 'file://' + HC.BASE_DIR + '/help/index.html' )
+            elif command == 'help': webbrowser.open( 'file://' + HC.HELP_DIR + '/index.html' )
             elif command == 'help_about': self._AboutWindow()
             elif command == 'help_shortcuts': wx.MessageBox( CC.SHORTCUT_HELP )
             elif command == 'import_files': self._ImportFiles()
@@ -2157,6 +2172,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
                 
                 HydrusGlobals.pubsub_profile_mode = not HydrusGlobals.pubsub_profile_mode
                 
+            elif command == 'rebalance_client_files': self._RebalanceClientFiles()
             elif command == 'redo': self._controller.pub( 'redo' )
             elif command == 'refresh':
                 
@@ -2178,10 +2194,6 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
                 if page is not None: page.ShowHideSplit()
                 
             elif command == 'site': webbrowser.open( 'https://hydrusnetwork.github.io/hydrus/' )
-            elif command == 'special_debug_mode':
-                
-                HydrusGlobals.special_debug_mode = not HydrusGlobals.special_debug_mode
-                
             elif command == 'start_url_download': self._StartURLDownload()
             elif command == 'start_youtube_download': self._StartYoutubeDownload()
             elif command == 'stats': self._Stats( data )

include/ClientGUICanvas.py

@@ -710,7 +710,9 @@ class Canvas( object ):
     
     def _CopyPathToClipboard( self ):
         
-        path = ClientFiles.GetFilePath( self._current_display_media.GetHash(), self._current_display_media.GetMime() )
+        client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+        
+        path = client_files_manager.GetFilePath( self._current_display_media.GetHash(), self._current_display_media.GetMime() )
         
         HydrusGlobals.client_controller.pub( 'clipboard', 'text', path )
         
@@ -861,7 +863,9 @@ class Canvas( object ):
             hash = self._current_display_media.GetHash()
             mime = self._current_display_media.GetMime()
             
-            path = ClientFiles.GetFilePath( hash, mime )
+            client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+            
+            path = client_files_manager.GetFilePath( hash, mime )
             
             HydrusPaths.LaunchFile( path )
             
@@ -2716,7 +2720,9 @@ class CanvasFullscreenMediaListCustomFilter( CanvasFullscreenMediaListNavigable
     
     def _CopyPathToClipboard( self ):
         
-        path = ClientFiles.GetFilePath( self._current_display_media.GetHash(), self._current_display_media.GetMime() )
+        client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+        
+        path = client_files_manager.GetFilePath( self._current_display_media.GetHash(), self._current_display_media.GetMime() )
         
         HydrusGlobals.client_controller.pub( 'clipboard', 'text', path )
         
@@ -3175,7 +3181,9 @@ class MediaContainer( wx.Window ):
                     
                     self._media_window = wx.lib.flashwin.FlashWindow( self, size = media_initial_size, pos = media_initial_position )
                     
-                    self._media_window.movie = ClientFiles.GetFilePath( self._media.GetHash(), HC.APPLICATION_FLASH )
+                    client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+                    
+                    self._media_window.movie = client_files_manager.GetFilePath( self._media.GetHash(), HC.APPLICATION_FLASH )
                     
                 else:
                     
@@ -3417,7 +3425,9 @@ class OpenExternallyButton( wx.Button ):
         hash = self._media.GetHash()
         mime = self._media.GetMime()
         
-        path = ClientFiles.GetFilePath( hash, mime )
+        client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+        
+        path = client_files_manager.GetFilePath( hash, mime )
         
         HydrusPaths.LaunchFile( path )
         

include/ClientGUICommon.py

@@ -145,6 +145,8 @@ class AutoCompleteDropdown( wx.Panel ):
         
         wx.Panel.__init__( self, parent )
         
+        self._intercept_key_events = True
+        
         self._last_search_text = ''
         self._next_updatelist_is_probably_fast = False
         
@@ -394,39 +396,87 @@ class AutoCompleteDropdown( wx.Panel ):
         
         HydrusGlobals.client_controller.ResetIdleTimer()
         
-        if event.KeyCode in ( wx.WXK_RETURN, wx.WXK_NUMPAD_ENTER ) and self._ShouldTakeResponsibilityForEnter():
+        if event.KeyCode in ( wx.WXK_TAB, wx.WXK_NUMPAD_TAB ):
             
-            self._TakeResponsibilityForEnter()
-            
-        elif event.KeyCode == wx.WXK_ESCAPE:
-            
-            self.GetTopLevelParent().SetFocus()
-            
-        elif event.KeyCode in ( wx.WXK_UP, wx.WXK_NUMPAD_UP, wx.WXK_DOWN, wx.WXK_NUMPAD_DOWN ) and self._text_ctrl.GetValue() == '' and len( self._dropdown_list ) == 0:
-            
-            if event.KeyCode in ( wx.WXK_UP, wx.WXK_NUMPAD_UP ): id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'select_up' )
-            elif event.KeyCode in ( wx.WXK_DOWN, wx.WXK_NUMPAD_DOWN ): id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'select_down' )
-            
-            new_event = wx.CommandEvent( commandType = wx.wxEVT_COMMAND_MENU_SELECTED, winid = id )
-            
-            self._text_ctrl.ProcessEvent( new_event )
-            
-        elif event.KeyCode in ( wx.WXK_PAGEDOWN, wx.WXK_NUMPAD_PAGEDOWN, wx.WXK_PAGEUP, wx.WXK_NUMPAD_PAGEUP ) and self._text_ctrl.GetValue() == '' and len( self._dropdown_list ) == 0:
-            
-            if event.KeyCode in ( wx.WXK_PAGEUP, wx.WXK_NUMPAD_PAGEUP ):
+            if event.ShiftDown():
                 
-                id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'canvas_show_previous' )
+                if self._intercept_key_events:
+                    
+                    self._intercept_key_events = False
+                    
+                    ( r, g, b ) = HC.options[ 'gui_colours' ][ 'autocomplete_background' ]
+                    
+                    if r != g or r != b or g != b:
+                        
+                        colour = wx.Colour( g, b, r )
+                        
+                    elif r > 127:
+                        
+                        colour = wx.Colour( g, b, r / 2 )
+                        
+                    else:
+                        
+                        colour = wx.Colour( g, b, r * 2 )
+                        
+                    
+                else:
+                    
+                    self._intercept_key_events = True
+                    
+                    colour = wx.Colour( *HC.options[ 'gui_colours' ][ 'autocomplete_background' ] )
+                    
                 
-            elif event.KeyCode in ( wx.WXK_PAGEDOWN, wx.WXK_NUMPAD_PAGEDOWN ):
+                self._text_ctrl.SetBackgroundColour( colour )
                 
-                id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'canvas_show_next' )
+                self._text_ctrl.Refresh()
+                
+            else:
+                
+                self._UpdateList()
+                
+                self._lag_timer.Stop()
                 
             
-            new_event = wx.CommandEvent( commandType = wx.wxEVT_COMMAND_MENU_SELECTED, winid = id )
+        elif self._intercept_key_events:
             
-            self._text_ctrl.ProcessEvent( new_event )
+            if event.KeyCode in ( wx.WXK_RETURN, wx.WXK_NUMPAD_ENTER ) and self._ShouldTakeResponsibilityForEnter():
+                
+                self._TakeResponsibilityForEnter()
+                
+            elif event.KeyCode == wx.WXK_ESCAPE:
+                
+                self.GetTopLevelParent().SetFocus()
+                
+            elif event.KeyCode in ( wx.WXK_UP, wx.WXK_NUMPAD_UP, wx.WXK_DOWN, wx.WXK_NUMPAD_DOWN ) and self._text_ctrl.GetValue() == '' and len( self._dropdown_list ) == 0:
+                
+                if event.KeyCode in ( wx.WXK_UP, wx.WXK_NUMPAD_UP ): id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'select_up' )
+                elif event.KeyCode in ( wx.WXK_DOWN, wx.WXK_NUMPAD_DOWN ): id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'select_down' )
+                
+                new_event = wx.CommandEvent( commandType = wx.wxEVT_COMMAND_MENU_SELECTED, winid = id )
+                
+                self._text_ctrl.ProcessEvent( new_event )
+                
+            elif event.KeyCode in ( wx.WXK_PAGEDOWN, wx.WXK_NUMPAD_PAGEDOWN, wx.WXK_PAGEUP, wx.WXK_NUMPAD_PAGEUP ) and self._text_ctrl.GetValue() == '' and len( self._dropdown_list ) == 0:
+                
+                if event.KeyCode in ( wx.WXK_PAGEUP, wx.WXK_NUMPAD_PAGEUP ):
+                    
+                    id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'canvas_show_previous' )
+                    
+                elif event.KeyCode in ( wx.WXK_PAGEDOWN, wx.WXK_NUMPAD_PAGEDOWN ):
+                    
+                    id = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'canvas_show_next' )
+                    
+                
+                new_event = wx.CommandEvent( commandType = wx.wxEVT_COMMAND_MENU_SELECTED, winid = id )
+                
+                self._text_ctrl.ProcessEvent( new_event )
+                
+            else: self._dropdown_list.ProcessEvent( event )
+            
+        else:
+            
+            event.Skip()
             
-        else: self._dropdown_list.ProcessEvent( event )
         
     
     def EventKillFocus( self, event ):
@@ -500,11 +550,20 @@ class AutoCompleteDropdown( wx.Panel ):
         
         num_chars = len( self._text_ctrl.GetValue() )
         
-        ( char_limit, long_wait, short_wait ) = HC.options[ 'ac_timings' ]
-        
-        if num_chars == 0 or self._next_updatelist_is_probably_fast: self._UpdateList()
-        elif num_chars < char_limit: self._lag_timer.Start( long_wait, wx.TIMER_ONE_SHOT )
-        else: self._lag_timer.Start( short_wait, wx.TIMER_ONE_SHOT )
+        if num_chars == 0:
+            
+            self._UpdateList()
+            
+        elif HC.options[ 'fetch_ac_results_automatically' ]:
+            
+            ( char_limit, long_wait, short_wait ) = HC.options[ 'ac_timings' ]
+            
+            self._next_updatelist_is_probably_fast = self._next_updatelist_is_probably_fast and num_chars > len( self._last_search_text )
+            
+            if self._next_updatelist_is_probably_fast: self._UpdateList()
+            elif num_chars < char_limit: self._lag_timer.Start( long_wait, wx.TIMER_ONE_SHOT )
+            else: self._lag_timer.Start( short_wait, wx.TIMER_ONE_SHOT )
+            
         
     
     def TIMEREventDropdownHide( self, event ):
@@ -680,7 +739,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
     
 class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
     
-    def __init__( self, parent, page_key, file_service_key, tag_service_key, media_callable = None, include_current = True, include_pending = True, synchronised = True ):
+    def __init__( self, parent, page_key, file_service_key, tag_service_key, media_callable = None, include_current = True, include_pending = True, synchronised = True, include_unusual_predicate_types = True ):
         
         AutoCompleteDropdownTags.__init__( self, parent, file_service_key, tag_service_key )
         
@@ -698,6 +757,8 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
         self._synchronised = OnOffButton( self._dropdown_window, self._page_key, 'notify_search_immediately', on_label = 'searching immediately', off_label = 'waiting -- tag counts may be inaccurate', start_on = synchronised )
         self._synchronised.SetToolTipString( 'select whether to renew the search as soon as a new predicate is entered' )
         
+        self._include_unusual_predicate_types = include_unusual_predicate_types
+        
         button_hbox_1 = wx.BoxSizer( wx.HORIZONTAL )
         
         button_hbox_1.AddF( self._include_current_tags, CC.FLAGS_EXPAND_BOTH_WAYS )
@@ -833,6 +894,8 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
                 half_complete_tag = search_text
                 
             
+            siblings_manager = HydrusGlobals.client_controller.GetManager( 'tag_siblings' )
+            
             if half_complete_tag == '':
                 
                 self._cache_text = self._current_namespace + ':'
@@ -858,7 +921,9 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
                         
                         predicates = HydrusGlobals.client_controller.Read( 'autocomplete_predicates', file_service_key = self._file_service_key, tag_service_key = self._tag_service_key, tag = search_text, include_current = self._include_current, include_pending = self._include_pending, add_namespaceless = True )
                         
-                        predicates = ClientSearch.SortPredicates( predicates, collapse_siblings = True )
+                        predicates = siblings_manager.CollapsePredicates( predicates )
+                        
+                        predicates = ClientSearch.SortPredicates( predicates )
                         
                     else:
                         
@@ -868,7 +933,9 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
                             
                             self._cached_results = HydrusGlobals.client_controller.Read( 'autocomplete_predicates', file_service_key = self._file_service_key, tag_service_key = self._tag_service_key, half_complete_tag = search_text, include_current = self._include_current, include_pending = self._include_pending, add_namespaceless = True )
                             
-                            self._cached_results = ClientSearch.SortPredicates( self._cached_results, collapse_siblings = False )
+                            self._cached_results = siblings_manager.CollapsePredicates( self._cached_results )
+                            
+                            self._cached_results = ClientSearch.SortPredicates( self._cached_results )
                             
                         
                         predicates = self._cached_results
@@ -908,7 +975,9 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
                     
                     predicates = [ ClientData.Predicate( HC.PREDICATE_TYPE_TAG, tag, inclusive = inclusive, counts = { HC.CURRENT : current_tags_to_count[ tag ], HC.PENDING : pending_tags_to_count[ tag ] } ) for tag in tags_to_do ]
                     
-                    predicates = ClientSearch.SortPredicates( predicates, collapse_siblings = True )
+                    predicates = siblings_manager.CollapsePredicates( predicates )
+                    
+                    predicates = ClientSearch.SortPredicates( predicates )
                     
                     self._next_updatelist_is_probably_fast = True
                     
@@ -916,24 +985,27 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
                 matches = ClientSearch.FilterPredicates( search_text, predicates )
                 
             
-            if self._current_namespace != '':
+            if self._include_unusual_predicate_types:
                 
-                if '*' not in self._current_namespace:
+                if self._current_namespace != '':
                     
-                    matches.insert( 0, ClientData.Predicate( HC.PREDICATE_TYPE_NAMESPACE, self._current_namespace, inclusive = inclusive ) )
-                    
-                
-                if half_complete_tag != '':
-                    
-                    if '*' in self._current_namespace or ( '*' in half_complete_tag and half_complete_tag != '*' ):
+                    if '*' not in self._current_namespace:
                         
-                        matches.insert( 0, ClientData.Predicate( HC.PREDICATE_TYPE_WILDCARD, search_text, inclusive = inclusive ) )
+                        matches.insert( 0, ClientData.Predicate( HC.PREDICATE_TYPE_NAMESPACE, self._current_namespace, inclusive = inclusive ) )
                         
                     
-                
-            elif '*' in search_text:
-                
-                matches.insert( 0, ClientData.Predicate( HC.PREDICATE_TYPE_WILDCARD, search_text, inclusive = inclusive ) )
+                    if half_complete_tag != '':
+                        
+                        if '*' in self._current_namespace or ( '*' in half_complete_tag and half_complete_tag != '*' ):
+                            
+                            matches.insert( 0, ClientData.Predicate( HC.PREDICATE_TYPE_WILDCARD, search_text, inclusive = inclusive ) )
+                            
+                        
+                    
+                elif '*' in search_text:
+                    
+                    matches.insert( 0, ClientData.Predicate( HC.PREDICATE_TYPE_WILDCARD, search_text, inclusive = inclusive ) )
+                    
                 
             
             try:
@@ -1144,7 +1216,7 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
                 
                 predicates = HydrusGlobals.client_controller.Read( 'autocomplete_predicates', file_service_key = self._file_service_key, tag_service_key = self._tag_service_key, tag = search_text, add_namespaceless = False )
                 
-                predicates = ClientSearch.SortPredicates( predicates, collapse_siblings = False )
+                predicates = ClientSearch.SortPredicates( predicates )
                 
             else:
                 
@@ -1154,7 +1226,7 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
                     
                     self._cached_results = HydrusGlobals.client_controller.Read( 'autocomplete_predicates', file_service_key = self._file_service_key, tag_service_key = self._tag_service_key, half_complete_tag = search_text, add_namespaceless = False )
                     
-                    self._cached_results = ClientSearch.SortPredicates( self._cached_results, collapse_siblings = False )
+                    self._cached_results = ClientSearch.SortPredicates( self._cached_results )
                     
                 
                 predicates = self._cached_results
@@ -1304,8 +1376,8 @@ class BufferedWindow( wx.Window ):
             
             self._dirty = True
             
-            self.Refresh()
-            
+        
+        self.Refresh()
         
     
 class BufferedWindowIcon( BufferedWindow ):
@@ -3746,12 +3818,13 @@ class ListCtrlAutoWidth( wx.ListCtrl, ListCtrlAutoWidthMixin ):
     
 class NoneableSpinCtrl( wx.Panel ):
     
-    def __init__( self, parent, message, none_phrase = 'no limit', min = 0, max = 1000000, multiplier = 1, num_dimensions = 1 ):
+    def __init__( self, parent, message, none_phrase = 'no limit', min = 0, max = 1000000, unit = None, multiplier = 1, num_dimensions = 1 ):
         
         wx.Panel.__init__( self, parent )
         
-        self._num_dimensions = num_dimensions
+        self._unit = unit
         self._multiplier = multiplier
+        self._num_dimensions = num_dimensions
         
         self._checkbox = wx.CheckBox( self, label = none_phrase )
         self._checkbox.Bind( wx.EVT_CHECKBOX, self.EventCheckBox )
@@ -3775,6 +3848,11 @@ class NoneableSpinCtrl( wx.Panel ):
             hbox.AddF( self._two, CC.FLAGS_MIXED )
             
         
+        if self._unit is not None:
+            
+            hbox.AddF( wx.StaticText( self, label = unit ), CC.FLAGS_MIXED )
+            
+        
         hbox.AddF( self._checkbox, CC.FLAGS_MIXED )
         
         self.SetSizer( hbox )
@@ -5767,4 +5845,58 @@ class ShowKeys( Frame ):
                 
             
         
-    
+    
+class WaitingPolitely( BufferedWindow ):
+    
+    def __init__( self, parent, page_key ):
+        
+        BufferedWindow.__init__( self, parent, size = ( 19, 19 ) )
+        
+        self._page_key = page_key
+        self._waiting = False
+        
+        HydrusGlobals.client_controller.sub( self, 'SetWaitingPolitely', 'waiting_politely' )
+        
+        self.SetWaitingPolitely( self._page_key, False )
+        
+    
+    def _Draw( self, dc ):
+        
+        dc.SetBackground( wx.Brush( wx.SystemSettings.GetColour( wx.SYS_COLOUR_BTNFACE ) ) )
+        
+        dc.Clear()
+        
+        if self._waiting:
+            
+            dc.SetBrush( wx.Brush( wx.Colour( 250, 190, 77 ) ) )
+            
+        else:
+            
+            dc.SetBrush( wx.Brush( wx.Colour( 77, 250, 144 ) ) )
+            
+        
+        dc.SetPen( wx.BLACK_PEN )
+        
+        dc.DrawCircle( 9, 9, 7 )
+        
+    
+    def SetWaitingPolitely( self, page_key, value ):
+        
+        if page_key == self._page_key:
+            
+            self._waiting = value
+            
+            if self._waiting:
+                
+                self.SetToolTipString( 'waiting before attempting another download' )
+                
+            else:
+                
+                self.SetToolTipString( 'ready to download' )
+                
+            
+            self._dirty = True
+            
+            self.Refresh()
+            
+        

include/ClientGUIDialogs.py

@@ -648,7 +648,7 @@ class DialogFirstStart( Dialog ):
         self._ok.SetForegroundColour( ( 0, 128, 0 ) )
         
         message1 = 'Hi, this looks like the first time you have started the hydrus client. Don\'t forget to check out the'
-        link = wx.HyperlinkCtrl( self, id = -1, label = 'help', url = 'file://' + HC.BASE_DIR + '/help/index.html' )
+        link = wx.HyperlinkCtrl( self, id = -1, label = 'help', url = 'file://' + HC.HELP_DIR + '/index.html' )
         message2 = 'if you haven\'t already.'
         message3 = 'When you close this dialog, the client will start its local http server. You will probably get a firewall warning.'
         message4 = 'You can block it if you like, or you can allow it. It doesn\'t phone home, or expose your files to your network; it just provides another way to locally export your files.'
@@ -4468,6 +4468,8 @@ class DialogSetupExport( Dialog ):
         
         terms = ClientFiles.ParseExportPhrase( pattern )
         
+        client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+        
         for ( ( ordering_index, media ), mime, path ) in self._paths.GetClientData():
             
             try:
@@ -4490,10 +4492,10 @@ class DialogSetupExport( Dialog ):
                         
                     
                 
-                source_path = ClientFiles.GetFilePath( hash, mime )
+                source_path = client_files_manager.GetFilePath( hash, mime )
+                
+                shutil.copy2( source_path, path )
                 
-                shutil.copy( source_path, path )
-                shutil.copystat( source_path, path )
                 try: os.chmod( path, stat.S_IWRITE | stat.S_IREAD )
                 except: pass
                 

include/ClientGUIDialogsManage.py

@@ -3069,6 +3069,7 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
         self._listbook.AddPage( 'local server', self._ServerPanel( self._listbook ) )
         self._listbook.AddPage( 'sort/collect', self._SortCollectPanel( self._listbook ) )
         self._listbook.AddPage( 'shortcuts', self._ShortcutsPanel( self._listbook ) )
+        self._listbook.AddPage( 'file storage locations', self._ClientFilesPanel( self._listbook ) )
         self._listbook.AddPage( 'downloading', self._DownloadingPanel( self._listbook ) )
         self._listbook.AddPage( 'tags', self._TagsPanel( self._listbook, self._new_options ) )
         
@@ -3106,7 +3107,136 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
         
         wx.CallAfter( self._ok.SetFocus )
         
-    
+
+    class _ClientFilesPanel( wx.Panel ):
+        
+        def __init__( self, parent ):
+            
+            wx.Panel.__init__( self, parent )
+            
+            self.SetBackgroundColour( wx.SystemSettings.GetColour( wx.SYS_COLOUR_BTNFACE ) )
+            
+            self._client_files = ClientGUICommon.SaneListCtrl( self, 200, [ ( 'path', -1 ), ( 'weight', 80 ) ] )
+            
+            self._add = wx.Button( self, label = 'add' )
+            self._add.Bind( wx.EVT_BUTTON, self.EventAdd )
+            
+            self._edit = wx.Button( self, label = 'edit weight' )
+            self._edit.Bind( wx.EVT_BUTTON, self.EventEditWeight )
+            
+            self._delete = wx.Button( self, label = 'delete' )
+            self._delete.Bind( wx.EVT_BUTTON, self.EventDelete )
+            
+            #
+            
+            self._new_options = HydrusGlobals.client_controller.GetNewOptions()
+            
+            for ( location, weight ) in self._new_options.GetClientFilesLocationsToIdealWeights().items():
+                
+                self._client_files.Append( ( location, HydrusData.ConvertIntToPrettyString( int( weight ) ) ), ( location, weight ) )
+                
+            
+            #
+            
+            vbox = wx.BoxSizer( wx.VERTICAL )
+            
+            text = 'Here you can change the folders where the client stores your files. Setting a higher weight increases the proportion of your collection that that folder stores.'
+            text += os.linesep * 2
+            text += 'If you add or remove folders here, it will take time for the client to incrementally rebalance your files across the new selection, but if you are in a hurry, you can force a full rebalance from the database->maintenance menu on the main gui.'
+            
+            st = wx.StaticText( self, label = text )
+            
+            st.Wrap( 400 )
+            
+            vbox.AddF( st, CC.FLAGS_EXPAND_PERPENDICULAR )
+            
+            vbox.AddF( self._client_files, CC.FLAGS_EXPAND_BOTH_WAYS )
+            
+            hbox = wx.BoxSizer( wx.HORIZONTAL )
+            
+            hbox.AddF( self._add, CC.FLAGS_MIXED )
+            hbox.AddF( self._edit, CC.FLAGS_MIXED )
+            hbox.AddF( self._delete, CC.FLAGS_MIXED )
+            
+            vbox.AddF( hbox, CC.FLAGS_BUTTON_SIZER )
+            
+            self.SetSizer( vbox )
+            
+        
+        def EventAdd( self, event ):
+            
+            with wx.DirDialog( self, 'Select the file location' ) as dlg:
+                
+                if dlg.ShowModal() == wx.ID_OK:
+                    
+                    path = HydrusData.ToUnicode( dlg.GetPath() )
+                    
+                    for ( location, weight ) in self._client_files.GetClientData():
+                        
+                        if path == location:
+                            
+                            wx.MessageBox( 'You already have that location entered!' )
+                            
+                            return
+                            
+                        
+                    
+                    with wx.NumberEntryDialog( self, 'Enter the weight of ' + path + '.', '', 'Enter Weight', value = 1, min = 1, max = 256 ) as dlg_num:
+                        
+                        if dlg_num.ShowModal() == wx.ID_OK:
+                            
+                            weight = dlg_num.GetValue()
+                            
+                            weight = float( weight )
+                            
+                            self._client_files.Append( ( path, HydrusData.ConvertIntToPrettyString( int( weight ) ) ), ( path, weight ) )
+                            
+                        
+                    
+                
+            
+        
+        def EventDelete( self, event ):
+            
+            if len( self._client_files.GetAllSelected() ) < self._client_files.GetItemCount():
+                
+                self._client_files.RemoveAllSelected()
+                
+            
+        
+        def EventEditWeight( self, event ):
+            
+            for i in self._client_files.GetAllSelected():
+                
+                ( location, weight ) = self._client_files.GetClientData( i )
+                
+                with wx.NumberEntryDialog( self, 'Enter the weight of ' + location + '.', '', 'Enter Weight', value = int( weight ), min = 1, max = 256 ) as dlg:
+                    
+                    if dlg.ShowModal() == wx.ID_OK:
+                        
+                        weight = dlg.GetValue()
+                        
+                        weight = float( weight )
+                        
+                        self._client_files.UpdateRow( i, ( location, HydrusData.ConvertIntToPrettyString( int( weight ) ) ), ( location, weight ) )
+                        
+                    
+                
+            
+        
+        def UpdateOptions( self ):
+            
+            locations_to_weights = {}
+            
+            for ( location, weight ) in self._client_files.GetClientData():
+                
+                locations_to_weights[ location ] = weight
+                
+            
+            self._new_options.SetClientFilesLocationsToIdealWeights( locations_to_weights )
+            
+        
+
     class _ColoursPanel( wx.Panel ):
         
         def __init__( self, parent ):
@@ -3474,13 +3604,16 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             
             self.SetBackgroundColour( wx.SystemSettings.GetColour( wx.SYS_COLOUR_BTNFACE ) )
             
-            self._idle_panel = ClientGUICommon.StaticBox( self, 'when to run big jobs' )
+            self._idle_panel = ClientGUICommon.StaticBox( self, 'when to run high cpu jobs' )
             self._maintenance_panel = ClientGUICommon.StaticBox( self, 'maintenance period' )
             self._processing_panel = ClientGUICommon.StaticBox( self, 'processing' )
             
-            self._idle_period = wx.SpinCtrl( self._idle_panel, min = 0, max = 1000 )
-            self._idle_mouse_period = wx.SpinCtrl( self._idle_panel, min = 0, max = 1000 )
-            self._idle_cpu_max = wx.SpinCtrl( self._idle_panel, min = 0, max = 100 )
+            self._idle_normal = wx.CheckBox( self._idle_panel )
+            self._idle_normal.Bind( wx.EVT_CHECKBOX, self.EventIdleNormal )
+            
+            self._idle_period = ClientGUICommon.NoneableSpinCtrl( self._idle_panel, '', min = 1, max = 1000, multiplier = 60, unit = 'minutes', none_phrase = 'ignore normal browsing' )
+            self._idle_mouse_period = ClientGUICommon.NoneableSpinCtrl( self._idle_panel, '', min = 1, max = 1000, multiplier = 60, unit = 'minutes', none_phrase = 'ignore mouse movements' )
+            self._idle_cpu_max = ClientGUICommon.NoneableSpinCtrl( self._idle_panel, '', min = 0, max = 99, unit = '%', none_phrase = 'ignore cpu usage' )
             
             self._idle_shutdown = ClientGUICommon.BetterChoice( self._idle_panel )
             
@@ -3489,22 +3622,25 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
                 self._idle_shutdown.Append( CC.idle_string_lookup[ idle_id ], idle_id )
                 
             
+            self._idle_shutdown.Bind( wx.EVT_CHOICE, self.EventIdleShutdown )
+            
             self._idle_shutdown_max_minutes = wx.SpinCtrl( self._idle_panel, min = 1, max = 1440 )
             
-            self._maintenance_vacuum_period = wx.SpinCtrl( self._maintenance_panel, min = 0, max = 365 )
+            self._maintenance_vacuum_period = ClientGUICommon.NoneableSpinCtrl( self._maintenance_panel, '', min = 1, max = 365, multiplier = 86400, none_phrase = 'do not automatically vacuum' )
             
             self._processing_phase = wx.SpinCtrl( self._processing_panel, min = 0, max = 100000 )
             self._processing_phase.SetToolTipString( 'how long this client will delay processing updates after they are due. useful if you have multiple clients and do not want them to process at the same time' )
             
             #
             
-            self._idle_period.SetValue( HC.options[ 'idle_period' ] / 60 )
-            self._idle_mouse_period.SetValue( HC.options[ 'idle_mouse_period' ] / 60 )
+            self._idle_normal.SetValue( HC.options[ 'idle_normal' ] )
+            self._idle_period.SetValue( HC.options[ 'idle_period' ] )
+            self._idle_mouse_period.SetValue( HC.options[ 'idle_mouse_period' ] )
             self._idle_cpu_max.SetValue( HC.options[ 'idle_cpu_max' ] )
             self._idle_shutdown.SelectClientData( HC.options[ 'idle_shutdown' ] )
             self._idle_shutdown_max_minutes.SetValue( HC.options[ 'idle_shutdown_max_minutes' ] )
             
-            self._maintenance_vacuum_period.SetValue( HC.options[ 'maintenance_vacuum_period' ] / 86400 )
+            self._maintenance_vacuum_period.SetValue( HC.options[ 'maintenance_vacuum_period' ] )
             
             self._processing_phase.SetValue( HC.options[ 'processing_phase' ] )
             
@@ -3514,13 +3650,16 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             
             gridbox.AddGrowableCol( 1, 1 )
             
-            gridbox.AddF( wx.StaticText( self._idle_panel, label = 'Minutes of general client inactivity until client can be considered idle: ' ), CC.FLAGS_MIXED )
+            gridbox.AddF( wx.StaticText( self._idle_panel, label = 'Run jobs when the client is not busy?: ' ), CC.FLAGS_MIXED )
+            gridbox.AddF( self._idle_normal, CC.FLAGS_MIXED )
+            
+            gridbox.AddF( wx.StaticText( self._idle_panel, label = 'Assume the client is busy if general browsing activity has occured in the past: ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._idle_period, CC.FLAGS_MIXED )
             
-            gridbox.AddF( wx.StaticText( self._idle_panel, label = 'Minutes of unmoving mouse until client can be considered idle: ' ), CC.FLAGS_MIXED )
+            gridbox.AddF( wx.StaticText( self._idle_panel, label = 'Assume the client is busy if the mouse has been moved in the past: ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._idle_mouse_period, CC.FLAGS_MIXED )
             
-            gridbox.AddF( wx.StaticText( self._idle_panel, label = 'Do not start a job if any CPU core has more than this percent usage: ' ), CC.FLAGS_MIXED )
+            gridbox.AddF( wx.StaticText( self._idle_panel, label = 'Assume the client is busy if any CPU core has recent average usage above: ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._idle_cpu_max, CC.FLAGS_MIXED )
             
             gridbox.AddF( wx.StaticText( self._idle_panel, label = 'Run jobs on shutdown?: ' ), CC.FLAGS_MIXED )
@@ -3529,13 +3668,13 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             gridbox.AddF( wx.StaticText( self._idle_panel, label = 'Max number of minutes to run shutdown jobs: ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._idle_shutdown_max_minutes, CC.FLAGS_MIXED )
             
-            text = 'CPU-heavy jobs like maintenance routines and repository synchronisation processing will only start by themselves when you are not using the client.'
+            text = 'CPU-heavy jobs like maintenance routines and repository synchronisation processing will stutter or lock up your gui, so they do not normally run when you are searching for and looking at files.'
             text += os.linesep * 2
-            text += 'If you set both client inactivy and unmoving mouse to 0, the client will never consider itself idle and you should set up shutdown processing.'
+            text += 'You can set these jobs to run only when the client is not busy, or only during shutdown, or neither, or both.'
             
             st = wx.StaticText( self._idle_panel, label = text )
             
-            st.Wrap( 400 )
+            st.Wrap( 500 )
             
             self._idle_panel.AddF( st, CC.FLAGS_EXPAND_PERPENDICULAR )
             self._idle_panel.AddF( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
@@ -3546,7 +3685,7 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             
             gridbox.AddGrowableCol( 1, 1 )
             
-            gridbox.AddF( wx.StaticText( self._maintenance_panel, label = 'Number of days to wait between vacuums (0 for never): ' ), CC.FLAGS_MIXED )
+            gridbox.AddF( wx.StaticText( self._maintenance_panel, label = 'Number of days to wait between vacuums: ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._maintenance_vacuum_period, CC.FLAGS_MIXED )
             
             self._maintenance_panel.AddF( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
@@ -3557,7 +3696,7 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             
             gridbox.AddGrowableCol( 1, 1 )
             
-            gridbox.AddF( wx.StaticText( self._processing_panel, label = 'Delay update processing by (s): ' ), CC.FLAGS_MIXED )
+            gridbox.AddF( wx.StaticText( self._processing_panel, label = 'Delay repository update processing by (s): ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._processing_phase, CC.FLAGS_MIXED )
             
             self._processing_panel.AddF( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
@@ -3572,16 +3711,60 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             
             self.SetSizer( vbox )
             
+            self._EnableDisableIdleNormal()
+            self._EnableDisableIdleShutdown()
+            
+        
+        def _EnableDisableIdleNormal( self ):
+            
+            if self._idle_normal.GetValue() == True:
+                
+                self._idle_period.Enable()
+                self._idle_mouse_period.Enable()
+                self._idle_cpu_max.Enable()
+                
+            else:
+                
+                self._idle_period.Disable()
+                self._idle_mouse_period.Disable()
+                self._idle_cpu_max.Disable()
+                
+            
+        
+        def _EnableDisableIdleShutdown( self ):
+            
+            if self._idle_shutdown.GetChoice() == CC.IDLE_NOT_ON_SHUTDOWN:
+                
+                self._idle_shutdown_max_minutes.Disable()
+                
+            else:
+                
+                self._idle_shutdown_max_minutes.Enable()
+                
+            
+        
+        def EventIdleNormal( self, event ):
+            
+            self._EnableDisableIdleNormal()
+            
+        
+        def EventIdleShutdown( self, event ):
+            
+            self._EnableDisableIdleShutdown()
+            
         
         def UpdateOptions( self ):
             
-            HC.options[ 'idle_period' ] = 60 * self._idle_period.GetValue()
-            HC.options[ 'idle_mouse_period' ] = 60 * self._idle_mouse_period.GetValue()
+            HC.options[ 'idle_normal' ] = self._idle_normal.GetValue()
+            
+            HC.options[ 'idle_period' ] = self._idle_period.GetValue()
+            HC.options[ 'idle_mouse_period' ] = self._idle_mouse_period.GetValue()
             HC.options[ 'idle_cpu_max' ] = self._idle_cpu_max.GetValue()
+            
             HC.options[ 'idle_shutdown' ] = self._idle_shutdown.GetChoice()
             HC.options[ 'idle_shutdown_max_minutes' ] = self._idle_shutdown_max_minutes.GetValue()
             
-            HC.options[ 'maintenance_vacuum_period' ] = 86400 * self._maintenance_vacuum_period.GetValue()
+            HC.options[ 'maintenance_vacuum_period' ] = self._maintenance_vacuum_period.GetValue()
             
             HC.options[ 'processing_phase' ] = self._processing_phase.GetValue()
             
@@ -4427,16 +4610,19 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             self._estimated_number_fullscreens = wx.StaticText( self, label = '' )
             
             self._num_autocomplete_chars = wx.SpinCtrl( self, min = 1, max = 100 )
-            self._num_autocomplete_chars.SetToolTipString( 'how many characters you enter before the gui fetches autocomplete results from the db' + os.linesep + 'increase this if you find autocomplete results are slow' )
+            self._num_autocomplete_chars.SetToolTipString( 'how many characters you enter before the gui fetches autocomplete results from the db. (otherwise, it will only fetch exact matches)' + os.linesep + 'increase this if you find autocomplete results are slow' )
+            
+            self._fetch_ac_results_automatically = wx.CheckBox( self )
+            self._fetch_ac_results_automatically.Bind( wx.EVT_CHECKBOX, self.EventFetchAuto )
             
             self._autocomplete_long_wait = wx.SpinCtrl( self, min = 0, max = 10000 )
-            self._autocomplete_long_wait.SetToolTipString( 'how long the gui will wait, after you enter a character, before it queries the db with what you have entered so far' )
+            self._autocomplete_long_wait.SetToolTipString( 'how long the gui will typically wait, after you enter a character, before it queries the db with what you have entered so far' )
             
             self._autocomplete_short_wait_chars = wx.SpinCtrl( self, min = 1, max = 100 )
             self._autocomplete_short_wait_chars.SetToolTipString( 'how many characters you enter before the gui starts waiting the short time before querying the db' )
             
             self._autocomplete_short_wait = wx.SpinCtrl( self, min = 0, max = 10000 )
-            self._autocomplete_short_wait.SetToolTipString( 'how long the gui will wait, after you enter a lot of characters, before it queries the db with what you have entered so far' )
+            self._autocomplete_short_wait.SetToolTipString( 'how long the gui will typically wait, after you enter a lot of characters, before it queries the db with what you have entered so far' )
             
             #
             
@@ -4454,6 +4640,8 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             
             self._num_autocomplete_chars.SetValue( HC.options[ 'num_autocomplete_chars' ] )
             
+            self._fetch_ac_results_automatically.SetValue( HC.options[ 'fetch_ac_results_automatically' ] )
+            
             ( char_limit, long_wait, short_wait ) = HC.options[ 'ac_timings' ]
             
             self._autocomplete_long_wait.SetValue( long_wait )
@@ -4479,6 +4667,8 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             fullscreens_sizer.AddF( self._fullscreen_cache_size, CC.FLAGS_MIXED )
             fullscreens_sizer.AddF( self._estimated_number_fullscreens, CC.FLAGS_MIXED )
             
+            vbox = wx.BoxSizer( wx.VERTICAL )
+            
             gridbox = wx.FlexGridSizer( 0, 2 )
             
             gridbox.AddGrowableCol( 1, 1 )
@@ -4489,6 +4679,12 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             gridbox.AddF( wx.StaticText( self, label = 'Thumbnail height: ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._thumbnail_height, CC.FLAGS_MIXED )
             
+            vbox.AddF( gridbox, CC.FLAGS_EXPAND_PERPENDICULAR )
+            
+            gridbox = wx.FlexGridSizer( 0, 2 )
+            
+            gridbox.AddGrowableCol( 1, 1 )
+            
             gridbox.AddF( wx.StaticText( self, label = 'MB memory reserved for thumbnail cache: ' ), CC.FLAGS_MIXED )
             gridbox.AddF( thumbnails_sizer, CC.FLAGS_NONE )
             
@@ -4498,9 +4694,22 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             gridbox.AddF( wx.StaticText( self, label = 'MB memory reserved for media viewer cache: ' ), CC.FLAGS_MIXED )
             gridbox.AddF( fullscreens_sizer, CC.FLAGS_NONE )
             
+            vbox.AddF( gridbox, CC.FLAGS_EXPAND_PERPENDICULAR )
+            
+            text = 'If you want to disable automatic autocomplete results fetching, use the Tab key to fetch results manually.'
+            
+            vbox.AddF( wx.StaticText( self, label = text ), CC.FLAGS_EXPAND_PERPENDICULAR )
+            
+            gridbox = wx.FlexGridSizer( 0, 2 )
+            
+            gridbox.AddGrowableCol( 1, 1 )
+            
             gridbox.AddF( wx.StaticText( self, label = 'Autocomplete character threshold: ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._num_autocomplete_chars, CC.FLAGS_MIXED )
             
+            gridbox.AddF( wx.StaticText( self, label = 'Automatically fetch autocomplete results after a short delay: ' ), CC.FLAGS_MIXED )
+            gridbox.AddF( self._fetch_ac_results_automatically, CC.FLAGS_MIXED )
+            
             gridbox.AddF( wx.StaticText( self, label = 'Autocomplete long wait (ms): ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._autocomplete_long_wait, CC.FLAGS_MIXED )
             
@@ -4510,10 +4719,13 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             gridbox.AddF( wx.StaticText( self, label = 'Autocomplete short wait (ms): ' ), CC.FLAGS_MIXED )
             gridbox.AddF( self._autocomplete_short_wait, CC.FLAGS_MIXED )
             
-            self.SetSizer( gridbox )
+            vbox.AddF( gridbox, CC.FLAGS_EXPAND_PERPENDICULAR )
+            
+            self.SetSizer( vbox )
             
             #
             
+            self.EventFetchAuto( None )
             self.EventFullscreensUpdate( None )
             self.EventPreviewsUpdate( None )
             self.EventThumbnailsUpdate( None )
@@ -4521,6 +4733,22 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             wx.CallAfter( self.Layout ) # draws the static texts correctly
             
         
+        def EventFetchAuto( self, event ):
+            
+            if self._fetch_ac_results_automatically.GetValue() == True:
+                
+                self._autocomplete_long_wait.Enable()
+                self._autocomplete_short_wait_chars.Enable()
+                self._autocomplete_short_wait.Enable()
+                
+            else:
+                
+                self._autocomplete_long_wait.Disable()
+                self._autocomplete_short_wait_chars.Disable()
+                self._autocomplete_short_wait.Disable()
+                
+            
+        
         def EventFullscreensUpdate( self, event ):
             
             ( width, height ) = wx.GetDisplaySize()
@@ -4566,6 +4794,8 @@ class DialogManageOptions( ClientGUIDialogs.Dialog ):
             
             HC.options[ 'num_autocomplete_chars' ] = self._num_autocomplete_chars.GetValue()
             
+            HC.options[ 'fetch_ac_results_automatically' ] = self._fetch_ac_results_automatically.GetValue()
+            
             long_wait = self._autocomplete_long_wait.GetValue()
             
             char_limit = self._autocomplete_short_wait_chars.GetValue()
@@ -6562,7 +6792,7 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
             
             ( service_key, service_type, name, info ) = self._original_info
             
-            message = 'This will completely reset ' + name + ', deleting all downloaded and processed information from the database. It may take several minutes to finish the operation, during which time the gui will likely freeze.' + os.linesep * 2 + 'Once the service is reset, the client will have to redownload and reprocess everything, which could take a very long time.' + os.linesep * 2 + 'If you do not understand what this button does, you definitely want to click no!'
+            message = 'This will completely reset ' + name + ', deleting all downloaded and processed information from the database. It may take several minutes to finish the operation, during which time the gui will likely freeze.' + os.linesep * 2 + 'Once the service is reset, the client will slowly redownload and reprocess everything in the background.' + os.linesep * 2 + 'If you do not understand what this button does, you definitely want to click no!'
             
             with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
                 
@@ -6579,7 +6809,7 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
             
             ( service_key, service_type, name, info ) = self._original_info
             
-            message = 'This will remove all the processed information for ' + name + ' from the database. It may take several minutes to finish the operation, during which time the gui will likely freeze.' + os.linesep * 2 + 'Once the service is reset, the client will have to reprocess all the downloaded updates, which could take a very long time.' + os.linesep * 2 + 'If you do not understand what this button does, you probably want to click no!'
+            message = 'This will remove all the processed information for ' + name + ' from the database. It may take several minutes to finish the operation, during which time the gui will likely freeze.' + os.linesep * 2 + 'Once the service is reset, the client will slowly reprocess everything in the background.' + os.linesep * 2 + 'If you do not understand what this button does, you probably want to click no!'
             
             with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
                 

include/ClientGUIHoverFrames.py

@@ -26,7 +26,7 @@ class FullscreenHoverFrame( wx.Frame ):
         self._canvas_key = canvas_key
         self._current_media = None
         
-        self._last_ideal_size_and_position = None
+        self._last_ideal_position = None
         
         self.SetBackgroundColour( wx.WHITE )
         self.SetCursor( wx.StockCursor( wx.CURSOR_ARROW ) )
@@ -51,20 +51,16 @@ class FullscreenHoverFrame( wx.Frame ):
         
         ( should_resize, my_ideal_size, my_ideal_position ) = self._GetIdealSizeAndPosition()
         
-        if should_resize or ( my_ideal_size, my_ideal_position ) != self._last_ideal_size_and_position:
+        if should_resize:
             
-            self._last_ideal_size_and_position = ( my_ideal_size, my_ideal_position )
+            self.Fit()
             
-            if should_resize:
-                
-                self.Fit()
-                
-                self.SetSize( my_ideal_size )
-                
-            
-            self.SetPosition( my_ideal_position )
+            self.SetSize( my_ideal_size )
             
         
+        self.SetPosition( my_ideal_position )
+        
+        
     
     def SetDisplayMedia( self, canvas_key, media ):
         

include/ClientGUIManagement.py

@@ -135,13 +135,13 @@ def CreateManagementControllerQuery( file_service_key, file_search_context, sear
     
     return management_controller
     
-def CreateManagementPanel( parent, page, management_controller ):
+def CreateManagementPanel( parent, page, controller, management_controller ):
     
     management_type = management_controller.GetType()
     
     management_class = management_panel_types_to_classes[ management_type ]
     
-    management_panel = management_class( parent, page, management_controller )
+    management_panel = management_class( parent, page, controller, management_controller )
     
     return management_panel
     
@@ -372,13 +372,13 @@ def GenerateDumpMultipartFormDataCTAndBody( fields ):
     
     def EventRefreshCaptcha( self, event ):
         
-        javascript_string = HydrusGlobals.client_controller.DoHTTP( HC.GET, 'http://www.google.com/recaptcha/api/challenge?k=' + self._captcha_key )
+        javascript_string = self._controller.DoHTTP( HC.GET, 'http://www.google.com/recaptcha/api/challenge?k=' + self._captcha_key )
         
         ( trash, rest ) = javascript_string.split( 'challenge : \'', 1 )
         
         ( self._captcha_challenge, trash ) = rest.split( '\'', 1 )
         
-        jpeg = HydrusGlobals.client_controller.DoHTTP( HC.GET, 'http://www.google.com/recaptcha/api/image?c=' + self._captcha_challenge )
+        jpeg = self._controller.DoHTTP( HC.GET, 'http://www.google.com/recaptcha/api/image?c=' + self._captcha_challenge )
         
         ( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
         
@@ -621,7 +621,7 @@ HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIAL
 
 class ManagementPanel( wx.lib.scrolledpanel.ScrolledPanel ):
     
-    def __init__( self, parent, page, management_controller ):
+    def __init__( self, parent, page, controller, management_controller ):
         
         wx.lib.scrolledpanel.ScrolledPanel.__init__( self, parent, style = wx.BORDER_NONE | wx.VSCROLL )
         
@@ -629,12 +629,13 @@ class ManagementPanel( wx.lib.scrolledpanel.ScrolledPanel ):
         
         self.SetBackgroundColour( wx.WHITE )
         
-        self._controller = management_controller
+        self._controller = controller
+        self._management_controller = management_controller
         
         self._page = page
-        self._page_key = self._controller.GetKey( 'page' )
+        self._page_key = self._management_controller.GetKey( 'page' )
         
-        HydrusGlobals.client_controller.sub( self, 'SetSearchFocus', 'set_search_focus' )
+        self._controller.sub( self, 'SetSearchFocus', 'set_search_focus' )
         
     
     def _MakeCollect( self, sizer ):
@@ -670,11 +671,11 @@ class ManagementPanel( wx.lib.scrolledpanel.ScrolledPanel ):
     '''
 class ManagementPanelDumper( ManagementPanel ):
     
-    def __init__( self, parent, page, management_controller ):
+    def __init__( self, parent, page, controller, management_controller ):
         
-        ManagementPanel.__init__( self, parent, page, management_controller )
+        ManagementPanel.__init__( self, parent, page, controller, management_controller )
         
-        ( self._4chan_token, pin, timeout ) = HydrusGlobals.client_controller.Read( '4chan_pass' )
+        ( self._4chan_token, pin, timeout ) = self._controller.Read( '4chan_pass' )
         
         self._have_4chan_pass = timeout > HydrusData.GetNow()
         
@@ -802,8 +803,8 @@ class ManagementPanelDumper( ManagementPanel ):
         
         self.SetSizer( vbox )
         
-        HydrusGlobals.client_controller.sub( self, 'FocusChanged', 'focus_changed' )
-        HydrusGlobals.client_controller.sub( self, 'SortedMediaPulse', 'sorted_media_pulse' )
+        self._controller.sub( self, 'FocusChanged', 'focus_changed' )
+        self._controller.sub( self, 'SortedMediaPulse', 'sorted_media_pulse' )
         
         self._sorted_media_hashes = [ media_result.GetHash() for media_result in media_results ]
         
@@ -841,7 +842,7 @@ class ManagementPanelDumper( ManagementPanel ):
         
         try:
             
-            response = HydrusGlobals.client_controller.DoHTTP( HC.POST, self._post_url, request_headers = headers, body = body )
+            response = self._controller.DoHTTP( HC.POST, self._post_url, request_headers = headers, body = body )
             
             ( status, phrase ) = ClientDownloading.Parse4chanPostScreen( response )
             
@@ -894,7 +895,7 @@ class ManagementPanelDumper( ManagementPanel ):
             
             tags_manager = media.GetTagsManager()
             
-            try: service = HydrusGlobals.client_controller.GetServicesManager().GetService( service_key )
+            try: service = self._controller.GetServicesManager().GetService( service_key )
             except HydrusExceptions.NotFoundException: continue
             
             service_key = service.GetServiceKey()
@@ -1028,7 +1029,7 @@ class ManagementPanelDumper( ManagementPanel ):
             dump_status_enum = CC.DUMPER_DUMPED_OK
             dump_status_string = 'dumped ok'
             
-            if hash == self._current_hash: HydrusGlobals.client_controller.pub( 'set_focus', self._page_key, None )
+            if hash == self._current_hash: self._controller.pub( 'set_focus', self._page_key, None )
             
             self._next_dump_time = HydrusData.GetNow() + self._flood_time
             
@@ -1100,7 +1101,7 @@ class ManagementPanelDumper( ManagementPanel ):
             dump_status_enum = CC.DUMPER_UNRECOVERABLE_ERROR
             dump_status_string = phrase
             
-            if hash == self._current_hash: HydrusGlobals.client_controller.pub( 'set_focus', self._page_key, None )
+            if hash == self._current_hash: self._controller.pub( 'set_focus', self._page_key, None )
             
             self._next_dump_time = HydrusData.GetNow() + self._flood_time
             
@@ -1109,7 +1110,7 @@ class ManagementPanelDumper( ManagementPanel ):
         
         self._hashes_to_dump_info[ hash ] = ( dump_status_enum, dump_status_string, post_field_info )
         
-        HydrusGlobals.client_controller.pub( 'file_dumped', self._page_key, hash, dump_status_enum )
+        self._controller.pub( 'file_dumped', self._page_key, hash, dump_status_enum )
         
         if self._next_dump_index == len( self._sorted_media_hashes ):
             
@@ -1262,7 +1263,7 @@ class ManagementPanelDumper( ManagementPanel ):
                 
                 media_to_select = self._hashes_to_media[ hash_to_select ]
                 
-                HydrusGlobals.client_controller.pub( 'set_focus', self._page_key, media_to_select )
+                self._controller.pub( 'set_focus', self._page_key, media_to_select )
                 
             
         
@@ -1363,7 +1364,9 @@ class ManagementPanelDumper( ManagementPanel ):
                         
                         mime = media.GetMime()
                         
-                        path = ClientFiles.GetFilePath( hash, mime )
+                        client_files_manager = self._controller.GetClientFilesManager()
+                        
+                        path = client_files_manager.GetFilePath( hash, mime )
                         
                         with open( path, 'rb' ) as f: file = f.read()
                         
@@ -1377,7 +1380,7 @@ class ManagementPanelDumper( ManagementPanel ):
                         
                         self._actually_dumping = True
                         
-                        HydrusGlobals.client_controller.CallToThread( self._THREADDoDump, hash, post_field_info, headers, body )
+                        self._controller.CallToThread( self._THREADDoDump, hash, post_field_info, headers, body )
                         
                     
                 except Exception as e:
@@ -1400,9 +1403,9 @@ management_panel_types_to_classes[ MANAGEMENT_TYPE_DUMPER ] = ManagementPanelDum
 '''
 class ManagementPanelGalleryImport( ManagementPanel ):
     
-    def __init__( self, parent, page, management_controller ):
+    def __init__( self, parent, page, controller, management_controller ):
         
-        ManagementPanel.__init__( self, parent, page, management_controller )
+        ManagementPanel.__init__( self, parent, page, controller, management_controller )
         
         self._gallery_downloader_panel = ClientGUICommon.StaticBox( self, 'gallery downloader' )
         
@@ -1413,6 +1416,8 @@ class ManagementPanelGalleryImport( ManagementPanel ):
         self._file_gauge = ClientGUICommon.Gauge( self._import_queue_panel )
         self._overall_gauge = ClientGUICommon.Gauge( self._import_queue_panel )
         
+        self._waiting_politely_indicator = ClientGUICommon.WaitingPolitely( self._import_queue_panel, self._page_key )
+        
         self._seed_cache_button = wx.BitmapButton( self._import_queue_panel, bitmap = CC.GlobalBMPs.seed_cache )
         self._seed_cache_button.Bind( wx.EVT_BUTTON, self.EventSeedCache )
         self._seed_cache_button.SetToolTipString( 'open detailed file import status' )
@@ -1487,6 +1492,7 @@ class ManagementPanelGalleryImport( ManagementPanel ):
         
         button_sizer = wx.BoxSizer( wx.HORIZONTAL )
         
+        button_sizer.AddF( self._waiting_politely_indicator, CC.FLAGS_MIXED )
         button_sizer.AddF( self._seed_cache_button, CC.FLAGS_MIXED )
         button_sizer.AddF( self._files_pause_button, CC.FLAGS_MIXED )
         
@@ -1520,9 +1526,9 @@ class ManagementPanelGalleryImport( ManagementPanel ):
         
         self.Bind( wx.EVT_MENU, self.EventMenu )
         
-        HydrusGlobals.client_controller.sub( self, 'UpdateStatus', 'update_status' )
+        self._controller.sub( self, 'UpdateStatus', 'update_status' )
         
-        self._gallery_import = self._controller.GetVariable( 'gallery_import' )
+        self._gallery_import = self._management_controller.GetVariable( 'gallery_import' )
         
         gallery_identifier = self._gallery_import.GetGalleryIdentifier()
         
@@ -1763,7 +1769,7 @@ class ManagementPanelGalleryImport( ManagementPanel ):
         
         seed_cache = self._gallery_import.GetSeedCache()
         
-        HydrusGlobals.client_controller.pub( 'show_seed_cache', seed_cache )
+        self._controller.pub( 'show_seed_cache', seed_cache )
         
     
     def SetSearchFocus( self, page_key ):
@@ -1783,9 +1789,9 @@ management_panel_types_to_classes[ MANAGEMENT_TYPE_IMPORT_GALLERY ] = Management
 
 class ManagementPanelHDDImport( ManagementPanel ):
     
-    def __init__( self, parent, page, management_controller ):
+    def __init__( self, parent, page, controller, management_controller ):
         
-        ManagementPanel.__init__( self, parent, page, management_controller )
+        ManagementPanel.__init__( self, parent, page, controller, management_controller )
         
         self._import_queue_panel = ClientGUICommon.StaticBox( self, 'import summary' )
         
@@ -1824,9 +1830,9 @@ class ManagementPanelHDDImport( ManagementPanel ):
         
         #
         
-        HydrusGlobals.client_controller.sub( self, 'UpdateStatus', 'update_status' )
+        self._controller.sub( self, 'UpdateStatus', 'update_status' )
         
-        self._hdd_import = self._controller.GetVariable( 'hdd_import' )
+        self._hdd_import = self._management_controller.GetVariable( 'hdd_import' )
         
         self._Update()
         
@@ -1904,7 +1910,7 @@ class ManagementPanelHDDImport( ManagementPanel ):
         
         seed_cache = self._hdd_import.GetSeedCache()
         
-        HydrusGlobals.client_controller.pub( 'show_seed_cache', seed_cache )
+        self._controller.pub( 'show_seed_cache', seed_cache )
         
     
     def TestAbleToClose( self ):
@@ -1935,9 +1941,9 @@ management_panel_types_to_classes[ MANAGEMENT_TYPE_IMPORT_HDD ] = ManagementPane
 
 class ManagementPanelPageOfImagesImport( ManagementPanel ):
     
-    def __init__( self, parent, page, management_controller ):
+    def __init__( self, parent, page, controller, management_controller ):
         
-        ManagementPanel.__init__( self, parent, page, management_controller )
+        ManagementPanel.__init__( self, parent, page, controller, management_controller )
         
         self._page_of_images_panel = ClientGUICommon.StaticBox( self, 'page of images downloader' )
         
@@ -1952,12 +1958,15 @@ class ManagementPanelPageOfImagesImport( ManagementPanel ):
         self._pause_button = wx.BitmapButton( self._import_queue_panel, bitmap = CC.GlobalBMPs.pause )
         self._pause_button.Bind( wx.EVT_BUTTON, self.EventPause )
         
+        self._waiting_politely_indicator = ClientGUICommon.WaitingPolitely( self._import_queue_panel, self._page_key )
+        
         self._seed_cache_button = wx.BitmapButton( self._import_queue_panel, bitmap = CC.GlobalBMPs.seed_cache )
         self._seed_cache_button.Bind( wx.EVT_BUTTON, self.EventSeedCache )
         self._seed_cache_button.SetToolTipString( 'open detailed file import status' )
         
         button_sizer = wx.BoxSizer( wx.HORIZONTAL )
         
+        button_sizer.AddF( self._waiting_politely_indicator, CC.FLAGS_MIXED )
         button_sizer.AddF( self._seed_cache_button, CC.FLAGS_MIXED )
         button_sizer.AddF( self._pause_button, CC.FLAGS_MIXED )
         
@@ -2034,9 +2043,9 @@ class ManagementPanelPageOfImagesImport( ManagementPanel ):
         
         self.Bind( wx.EVT_MENU, self.EventMenu )
         
-        HydrusGlobals.client_controller.sub( self, 'UpdateStatus', 'update_status' )
+        self._controller.sub( self, 'UpdateStatus', 'update_status' )
         
-        self._page_of_images_import = self._controller.GetVariable( 'page_of_images_import' )
+        self._page_of_images_import = self._management_controller.GetVariable( 'page_of_images_import' )
         
         def file_download_hook( gauge_range, gauge_value ):
             
@@ -2225,7 +2234,7 @@ class ManagementPanelPageOfImagesImport( ManagementPanel ):
         
         seed_cache = self._page_of_images_import.GetSeedCache()
         
-        HydrusGlobals.client_controller.pub( 'show_seed_cache', seed_cache )
+        self._controller.pub( 'show_seed_cache', seed_cache )
         
     
     def SetSearchFocus( self, page_key ):
@@ -2245,13 +2254,13 @@ management_panel_types_to_classes[ MANAGEMENT_TYPE_IMPORT_PAGE_OF_IMAGES ] = Man
 
 class ManagementPanelPetitions( ManagementPanel ):
     
-    def __init__( self, parent, page, management_controller ):
+    def __init__( self, parent, page, controller, management_controller ):
         
         self._petition_service_key = management_controller.GetKey( 'petition_service' )
         
-        ManagementPanel.__init__( self, parent, page, management_controller )
+        ManagementPanel.__init__( self, parent, page, controller, management_controller )
         
-        self._service = HydrusGlobals.client_controller.GetServicesManager().GetService( self._petition_service_key )
+        self._service = self._controller.GetServicesManager().GetService( self._petition_service_key )
         self._can_ban = self._service.GetInfo( 'account' ).HasPermission( HC.MANAGE_USERS )
         
         self._num_petitions = None
@@ -2321,7 +2330,7 @@ class ManagementPanelPetitions( ManagementPanel ):
         
         wx.CallAfter( self.EventRefreshNumPetitions, None )
         
-        HydrusGlobals.client_controller.sub( self, 'RefreshQuery', 'refresh_query' )
+        self._controller.sub( self, 'RefreshQuery', 'refresh_query' )
         
     
     def _DrawCurrentPetition( self ):
@@ -2375,9 +2384,9 @@ class ManagementPanelPetitions( ManagementPanel ):
     
     def _ShowHashes( self, hashes ):
         
-        file_service_key = self._controller.GetKey( 'file_service' )
+        file_service_key = self._management_controller.GetKey( 'file_service' )
     
-        with wx.BusyCursor(): media_results = HydrusGlobals.client_controller.Read( 'media_results', file_service_key, hashes )
+        with wx.BusyCursor(): media_results = self._controller.Read( 'media_results', file_service_key, hashes )
         
         panel = ClientGUIMedia.MediaPanelThumbnails( self._page, self._page_key, file_service_key, media_results )
         
@@ -2385,7 +2394,7 @@ class ManagementPanelPetitions( ManagementPanel ):
         
         panel.Sort( self._page_key, self._sort_by.GetChoice() )
         
-        HydrusGlobals.client_controller.pub( 'swap_media_panel', self._page_key, panel )
+        self._controller.pub( 'swap_media_panel', self._page_key, panel )
         
     
     def _DrawNumPetitions( self ):
@@ -2436,7 +2445,7 @@ class ManagementPanelPetitions( ManagementPanel ):
             
             self._service.Request( HC.POST, 'content_update_package', { 'update' : update } )
             
-            HydrusGlobals.client_controller.Write( 'content_updates', { self._petition_service_key : update.GetContentUpdates( for_client = True ) } )
+            self._controller.Write( 'content_updates', { self._petition_service_key : update.GetContentUpdates( for_client = True ) } )
             
         
         if len( denied_contents ) > 0:
@@ -2445,7 +2454,7 @@ class ManagementPanelPetitions( ManagementPanel ):
             
             self._service.Request( HC.POST, 'content_update_package', { 'update' : update } )
             
-            HydrusGlobals.client_controller.Write( 'content_updates', { self._petition_service_key : update.GetContentUpdates( for_client = True ) } )
+            self._controller.Write( 'content_updates', { self._petition_service_key : update.GetContentUpdates( for_client = True ) } )
             
         
         self._current_petition = None
@@ -2468,7 +2477,7 @@ class ManagementPanelPetitions( ManagementPanel ):
         
         self._DrawCurrentPetition()
         
-        HydrusGlobals.client_controller.CallToThread( do_it )
+        self._controller.CallToThread( do_it )
         
     
     def EventModifyPetitioner( self, event ):
@@ -2503,7 +2512,7 @@ class ManagementPanelPetitions( ManagementPanel ):
         
         self._num_petitions_text.SetLabel( u'Fetching\u2026' )
         
-        HydrusGlobals.client_controller.CallToThread( do_it )
+        self._controller.CallToThread( do_it )
         
     
     def RefreshQuery( self, page_key ):
@@ -2515,13 +2524,13 @@ management_panel_types_to_classes[ MANAGEMENT_TYPE_PETITIONS ] = ManagementPanel
 
 class ManagementPanelQuery( ManagementPanel ):
     
-    def __init__( self, parent, page, management_controller ):
+    def __init__( self, parent, page, controller, management_controller ):
         
-        ManagementPanel.__init__( self, parent, page, management_controller )
+        ManagementPanel.__init__( self, parent, page, controller, management_controller )
         
-        file_search_context = self._controller.GetVariable( 'file_search_context' )
+        file_search_context = self._management_controller.GetVariable( 'file_search_context' )
         
-        search_enabled = self._controller.GetVariable( 'search_enabled' )
+        search_enabled = self._management_controller.GetVariable( 'search_enabled' )
         
         self._query_key = HydrusThreading.JobKey( cancellable = True )
         
@@ -2533,12 +2542,12 @@ class ManagementPanelQuery( ManagementPanel ):
             
             self._current_predicates_box = ClientGUICommon.ListBoxTagsPredicates( self._search_panel, self._page_key, initial_predicates )
             
-            file_service_key = self._controller.GetKey( 'file_service' )
-            tag_service_key = self._controller.GetKey( 'tag_service' )
+            file_service_key = self._management_controller.GetKey( 'file_service' )
+            tag_service_key = self._management_controller.GetKey( 'tag_service' )
             
-            include_current = self._controller.GetVariable( 'include_current' )
-            include_pending = self._controller.GetVariable( 'include_pending' )
-            synchronised = self._controller.GetVariable( 'synchronised' )
+            include_current = self._management_controller.GetVariable( 'include_current' )
+            include_pending = self._management_controller.GetVariable( 'include_pending' )
+            synchronised = self._management_controller.GetVariable( 'synchronised' )
             
             self._searchbox = ClientGUICommon.AutoCompleteDropdownTagsRead( self._search_panel, self._page_key, file_service_key, tag_service_key, self._page.GetMedia, include_current = include_current, include_pending = include_pending, synchronised = synchronised )            
             self._search_panel.AddF( self._current_predicates_box, CC.FLAGS_EXPAND_PERPENDICULAR )
@@ -2558,43 +2567,43 @@ class ManagementPanelQuery( ManagementPanel ):
         
         if len( initial_predicates ) > 0 and not file_search_context.IsComplete(): wx.CallAfter( self._DoQuery )
         
-        HydrusGlobals.client_controller.sub( self, 'AddMediaResultsFromQuery', 'add_media_results_from_query' )
-        HydrusGlobals.client_controller.sub( self, 'ChangeFileRepositoryPubsub', 'change_file_repository' )
-        HydrusGlobals.client_controller.sub( self, 'ChangeTagRepositoryPubsub', 'change_tag_repository' )
-        HydrusGlobals.client_controller.sub( self, 'IncludeCurrent', 'notify_include_current' )
-        HydrusGlobals.client_controller.sub( self, 'IncludePending', 'notify_include_pending' )
-        HydrusGlobals.client_controller.sub( self, 'SearchImmediately', 'notify_search_immediately' )
-        HydrusGlobals.client_controller.sub( self, 'ShowQuery', 'file_query_done' )
-        HydrusGlobals.client_controller.sub( self, 'RefreshQuery', 'refresh_query' )
+        self._controller.sub( self, 'AddMediaResultsFromQuery', 'add_media_results_from_query' )
+        self._controller.sub( self, 'ChangeFileRepositoryPubsub', 'change_file_repository' )
+        self._controller.sub( self, 'ChangeTagRepositoryPubsub', 'change_tag_repository' )
+        self._controller.sub( self, 'IncludeCurrent', 'notify_include_current' )
+        self._controller.sub( self, 'IncludePending', 'notify_include_pending' )
+        self._controller.sub( self, 'SearchImmediately', 'notify_search_immediately' )
+        self._controller.sub( self, 'ShowQuery', 'file_query_done' )
+        self._controller.sub( self, 'RefreshQuery', 'refresh_query' )
         
     
     def _DoQuery( self ):
         
-        HydrusGlobals.client_controller.ResetIdleTimer()
+        self._controller.ResetIdleTimer()
         
         self._query_key.Cancel()
         
         self._query_key = HydrusThreading.JobKey()
         
-        if self._controller.GetVariable( 'search_enabled' ) and self._controller.GetVariable( 'synchronised' ):
+        if self._management_controller.GetVariable( 'search_enabled' ) and self._management_controller.GetVariable( 'synchronised' ):
             
             try:
                 
-                file_service_key = self._controller.GetKey( 'file_service' )
-                tag_service_key = self._controller.GetKey( 'tag_service' )
+                file_service_key = self._management_controller.GetKey( 'file_service' )
+                tag_service_key = self._management_controller.GetKey( 'tag_service' )
             
-                include_current = self._controller.GetVariable( 'include_current' )
-                include_pending = self._controller.GetVariable( 'include_pending' )
+                include_current = self._management_controller.GetVariable( 'include_current' )
+                include_pending = self._management_controller.GetVariable( 'include_pending' )
                 
                 current_predicates = self._current_predicates_box.GetPredicates()
                 
                 search_context = ClientData.FileSearchContext( file_service_key, tag_service_key, include_current, include_pending, current_predicates )
                 
-                self._controller.SetVariable( 'file_search_context', search_context )
+                self._management_controller.SetVariable( 'file_search_context', search_context )
                 
                 if len( current_predicates ) > 0:
                     
-                    HydrusGlobals.client_controller.StartFileQuery( self._query_key, search_context )
+                    self._controller.StartFileQuery( self._query_key, search_context )
                     
                     panel = ClientGUIMedia.MediaPanelLoading( self._page, self._page_key, file_service_key )
                     
@@ -2603,7 +2612,7 @@ class ManagementPanelQuery( ManagementPanel ):
                     panel = ClientGUIMedia.MediaPanelThumbnails( self._page, self._page_key, file_service_key, [] )
                     
                 
-                HydrusGlobals.client_controller.pub( 'swap_media_panel', self._page_key, panel )
+                self._controller.pub( 'swap_media_panel', self._page_key, panel )
                 
             except: wx.MessageBox( traceback.format_exc() )
             
@@ -2611,14 +2620,14 @@ class ManagementPanelQuery( ManagementPanel ):
     
     def AddMediaResultsFromQuery( self, query_key, media_results ):
         
-        if query_key == self._query_key: HydrusGlobals.client_controller.pub( 'add_media_results', self._page_key, media_results, append = False )
+        if query_key == self._query_key: self._controller.pub( 'add_media_results', self._page_key, media_results, append = False )
         
     
     def ChangeFileRepositoryPubsub( self, page_key, service_key ):
         
         if page_key == self._page_key:
             
-            self._controller.SetKey( 'file_service', service_key )
+            self._management_controller.SetKey( 'file_service', service_key )
             
             self._DoQuery()
             
@@ -2628,7 +2637,7 @@ class ManagementPanelQuery( ManagementPanel ):
         
         if page_key == self._page_key:
             
-            self._controller.SetKey( 'tag_service', service_key )
+            self._management_controller.SetKey( 'tag_service', service_key )
             
             self._DoQuery()
             
@@ -2651,7 +2660,7 @@ class ManagementPanelQuery( ManagementPanel ):
         
         if page_key == self._page_key:
             
-            self._controller.SetVariable( 'include_current', value )
+            self._management_controller.SetVariable( 'include_current', value )
             
             self._DoQuery()
             
@@ -2661,7 +2670,7 @@ class ManagementPanelQuery( ManagementPanel ):
         
         if page_key == self._page_key:
             
-            self._controller.SetVariable( 'include_pending', value )
+            self._management_controller.SetVariable( 'include_pending', value )
             
             self._DoQuery()
             
@@ -2676,7 +2685,7 @@ class ManagementPanelQuery( ManagementPanel ):
         
         if page_key == self._page_key:
             
-            self._controller.SetVariable( 'synchronised', value )
+            self._management_controller.SetVariable( 'synchronised', value )
             
             self._DoQuery()
             
@@ -2687,7 +2696,7 @@ class ManagementPanelQuery( ManagementPanel ):
         if page_key == self._page_key:
             
             try: self._searchbox.SetFocus() # there's a chance this doesn't exist!
-            except: HydrusGlobals.client_controller.pub( 'set_media_focus' )
+            except: self._controller.pub( 'set_media_focus' )
             
         
     
@@ -2699,7 +2708,7 @@ class ManagementPanelQuery( ManagementPanel ):
                 
                 current_predicates = self._current_predicates_box.GetPredicates()
                 
-                file_service_key = self._controller.GetKey( 'file_service' )
+                file_service_key = self._management_controller.GetKey( 'file_service' )
                 
                 panel = ClientGUIMedia.MediaPanelThumbnails( self._page, self._page_key, file_service_key, media_results )
                 
@@ -2707,7 +2716,7 @@ class ManagementPanelQuery( ManagementPanel ):
                 
                 panel.Sort( self._page_key, self._sort_by.GetChoice() )
                 
-                HydrusGlobals.client_controller.pub( 'swap_media_panel', self._page_key, panel )
+                self._controller.pub( 'swap_media_panel', self._page_key, panel )
                 
             
         except: wx.MessageBox( traceback.format_exc() )
@@ -2717,9 +2726,9 @@ management_panel_types_to_classes[ MANAGEMENT_TYPE_QUERY ] = ManagementPanelQuer
 
 class ManagementPanelThreadWatcherImport( ManagementPanel ):
     
-    def __init__( self, parent, page, management_controller ):
+    def __init__( self, parent, page, controller, management_controller ):
         
-        ManagementPanel.__init__( self, parent, page, management_controller )
+        ManagementPanel.__init__( self, parent, page, controller, management_controller )
         
         self._thread_watcher_panel = ClientGUICommon.StaticBox( self, 'thread watcher' )
         
@@ -2747,6 +2756,8 @@ class ManagementPanelThreadWatcherImport( ManagementPanel ):
         self._thread_check_now_button = wx.Button( self._options_panel, label = 'check now' )
         self._thread_check_now_button.Bind( wx.EVT_BUTTON, self.EventCheckNow )
         
+        self._waiting_politely_indicator = ClientGUICommon.WaitingPolitely( self._options_panel, self._page_key )
+        
         self._seed_cache_button = wx.BitmapButton( self._options_panel, bitmap = CC.GlobalBMPs.seed_cache )
         self._seed_cache_button.Bind( wx.EVT_BUTTON, self.EventSeedCache )
         self._seed_cache_button.SetToolTipString( 'open detailed file import status' )
@@ -2774,6 +2785,7 @@ class ManagementPanelThreadWatcherImport( ManagementPanel ):
         button_sizer = wx.BoxSizer( wx.HORIZONTAL )
         
         button_sizer.AddF( self._thread_check_now_button, CC.FLAGS_MIXED )
+        button_sizer.AddF( self._waiting_politely_indicator, CC.FLAGS_MIXED )
         button_sizer.AddF( self._seed_cache_button, CC.FLAGS_MIXED )
         button_sizer.AddF( self._pause_button, CC.FLAGS_MIXED )
         
@@ -2811,10 +2823,10 @@ class ManagementPanelThreadWatcherImport( ManagementPanel ):
         
         self.Bind( wx.EVT_MENU, self.EventMenu )
         
-        HydrusGlobals.client_controller.sub( self, 'UpdateStatus', 'update_status' )
-        HydrusGlobals.client_controller.sub( self, 'DecrementTimesToCheck', 'decrement_times_to_check' )
+        self._controller.sub( self, 'UpdateStatus', 'update_status' )
+        self._controller.sub( self, 'DecrementTimesToCheck', 'decrement_times_to_check' )
         
-        self._thread_watcher_import = self._controller.GetVariable( 'thread_watcher_import' )
+        self._thread_watcher_import = self._management_controller.GetVariable( 'thread_watcher_import' )
         
         def file_download_hook( gauge_range, gauge_value ):
             
@@ -3019,7 +3031,7 @@ class ManagementPanelThreadWatcherImport( ManagementPanel ):
         
         seed_cache = self._thread_watcher_import.GetSeedCache()
         
-        HydrusGlobals.client_controller.pub( 'show_seed_cache', seed_cache )
+        self._controller.pub( 'show_seed_cache', seed_cache )
         
     
     def EventTimesToCheck( self, event ):

include/ClientGUIMedia.py

@@ -191,7 +191,9 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
         
         display_media = self._focussed_media.GetDisplayMedia()
         
-        path = ClientFiles.GetFilePath( display_media.GetHash(), display_media.GetMime() )
+        client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+        
+        path = client_files_manager.GetFilePath( display_media.GetHash(), display_media.GetMime() )
         
         HydrusGlobals.client_controller.pub( 'clipboard', 'text', path )
         
@@ -330,7 +332,9 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
                 hash = display_media.GetHash()
                 mime = display_media.GetMime()
                 
-                path = ClientFiles.GetFilePath( hash, mime )
+                client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+                
+                path = client_files_manager.GetFilePath( hash, mime )
                 
                 HydrusPaths.LaunchFile( path )
                 
@@ -586,14 +590,19 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
         
         if self._focussed_media is not None:
             
-            hash = self._focussed_media.GetHash()
-            mime = self._focussed_media.GetMime()
-            
-            path = ClientFiles.GetFilePath( hash, mime )
-            
-            self._SetFocussedMedia( None )
-            
-            HydrusPaths.LaunchFile( path )
+            if CC.LOCAL_FILE_SERVICE_KEY in self._focussed_media.GetLocationsManager().GetCurrent():
+                
+                hash = self._focussed_media.GetHash()
+                mime = self._focussed_media.GetMime()
+                
+                client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+                
+                path = client_files_manager.GetFilePath( hash, mime )
+                
+                self._SetFocussedMedia( None )
+                
+                HydrusPaths.LaunchFile( path )
+                
             
         
     
@@ -1620,9 +1629,11 @@ class MediaPanelThumbnails( MediaPanel ):
                     
                     file_data_object = wx.FileDataObject()
                     
+                    client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+                    
                     for hash in hashes:
                         
-                        path = ClientFiles.GetFilePath( hash )
+                        path = client_files_manager.GetFilePath( hash )
                         
                         file_data_object.AddFile( path )
                         
@@ -2196,7 +2207,10 @@ class MediaPanelThumbnails( MediaPanel ):
                 
                 menu.AppendSeparator()
                 
-                menu.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'open_externally' ), '&open externally' )
+                if selection_has_local_file_service:
+                    
+                    menu.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'open_externally' ), '&open externally' )
+                    
                 
                 share_menu = wx.Menu()
                 

include/ClientGUIPages.py

@@ -45,7 +45,7 @@ class Page( wx.SplitterWindow ):
         
         self._search_preview_split.Bind( wx.EVT_SPLITTER_DCLICK, self.EventPreviewUnsplit )
         
-        self._management_panel = ClientGUIManagement.CreateManagementPanel( self._search_preview_split, self, self._management_controller )
+        self._management_panel = ClientGUIManagement.CreateManagementPanel( self._search_preview_split, self, self._controller, self._management_controller )
         
         file_service_key = self._management_controller.GetKey( 'file_service' )
         

include/ClientGUIPredicates.py

@@ -440,7 +440,7 @@ class PanelPredicateSystemRating( PanelPredicateSystem ):
         
         #
         
-        local_like_services = HydrusGlobals.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_LIKE, ) )
+        local_like_services = HydrusGlobals.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_LIKE, ), randomised = False )
         
         self._like_checkboxes_to_info = {}
         
@@ -471,7 +471,7 @@ class PanelPredicateSystemRating( PanelPredicateSystem ):
         
         #
         
-        local_numerical_services = HydrusGlobals.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_NUMERICAL, ) )
+        local_numerical_services = HydrusGlobals.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_NUMERICAL, ), randomised = False )
         
         self._numerical_checkboxes_to_info = {}
         

include/ClientImporting.py

@@ -245,7 +245,7 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
         
         if do_wait:
             
-            time.sleep( HC.options[ 'website_download_polite_wait' ] )
+            ClientData.WaitPolitely( page_key )
             
         
         with self._lock:
@@ -375,12 +375,7 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
             time.sleep( 5 )
             
         
-        with self._lock:
-            
-            self._SetGalleryStatus( page_key, 'waiting politely' )
-            
-        
-        time.sleep( HC.options[ 'website_download_polite_wait' ] )
+        ClientData.WaitPolitely( page_key )
         
         with self._lock:
             
@@ -1260,7 +1255,7 @@ class PageOfImagesImport( HydrusSerialisable.SerialisableBase ):
         
         if do_wait:
             
-            time.sleep( HC.options[ 'website_download_polite_wait' ] )
+            ClientData.WaitPolitely( page_key )
             
         
     
@@ -1358,12 +1353,7 @@ class PageOfImagesImport( HydrusSerialisable.SerialisableBase ):
             
             if not error_occurred and do_wait:
                 
-                with self._lock:
-                    
-                    self._SetParserStatus( page_key, 'waiting politely' )
-                    
-                
-                time.sleep( HC.options[ 'website_download_polite_wait' ] )
+                ClientData.WaitPolitely( page_key )
                 
             
             with self._lock:
@@ -2039,7 +2029,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
             
             if do_wait:
                 
-                time.sleep( HC.options[ 'website_download_polite_wait' ] )
+                ClientData.WaitPolitely()
                 
             
         
@@ -2102,7 +2092,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
                     
                     ( page_of_urls, definitely_no_more_pages ) = gallery.GetPage( self._query, page_index )
                     
-                    time.sleep( HC.options[ 'website_download_polite_wait' ] )
+                    ClientData.WaitPolitely()
                     
                     page_index += 1
                     
@@ -2463,7 +2453,7 @@ class ThreadWatcherImport( HydrusSerialisable.SerialisableBase ):
         
         if do_wait:
             
-            time.sleep( HC.options[ 'website_download_polite_wait' ] )
+            ClientData.WaitPolitely( page_key )
             
         
     
@@ -2614,12 +2604,7 @@ class ThreadWatcherImport( HydrusSerialisable.SerialisableBase ):
         
         if not error_occurred and do_wait:
             
-            with self._lock:
-                
-                self._SetWatcherStatus( page_key, 'waiting politely' )
-                
-            
-            time.sleep( HC.options[ 'website_download_polite_wait' ] )
+            ClientData.WaitPolitely( page_key )
             
         
         with self._lock:

include/ClientLocalServer.py

@@ -1,5 +1,4 @@
 import ClientLocalServerResources
-import HydrusServerResources
 import HydrusServer
 
 class HydrusServiceBooru( HydrusServer.HydrusService ):
@@ -12,7 +11,7 @@ class HydrusServiceBooru( HydrusServer.HydrusService ):
         root.putChild( 'page', ClientLocalServerResources.HydrusResourceCommandBooruPage( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
         root.putChild( 'file', ClientLocalServerResources.HydrusResourceCommandBooruFile( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
         root.putChild( 'thumbnail', ClientLocalServerResources.HydrusResourceCommandBooruThumbnail( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
-        root.putChild( 'style.css', HydrusServerResources.local_booru_css )
+        root.putChild( 'style.css', ClientLocalServerResources.local_booru_css )
         
         return root
 

include/ClientLocalServerResources.py

@@ -5,7 +5,9 @@ import HydrusData
 import HydrusGlobals
 import HydrusServerResources
 import os
-import wx
+from twisted.web.static import File as FileResource
+
+local_booru_css = FileResource( os.path.join( HC.STATIC_DIR, 'local_booru_style.css' ), defaultType = 'text/css' )
 
 class HydrusResourceCommandBooru( HydrusServerResources.HydrusResourceCommand ):
     
@@ -46,7 +48,9 @@ class HydrusResourceCommandBooruFile( HydrusResourceCommandBooru ):
         
         local_booru_manager.CheckFileAuthorised( share_key, hash )
         
-        path = ClientFiles.GetFilePath( hash )
+        client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+        
+        path = client_files_manager.GetFilePath( hash )
         
         response_context = HydrusServerResources.ResponseContext( 200, path = path )
         
@@ -242,11 +246,9 @@ class HydrusResourceCommandBooruThumbnail( HydrusResourceCommandBooru ):
         mime = media_result.GetMime()
         
         if mime in HC.MIMES_WITH_THUMBNAILS: path = ClientFiles.GetThumbnailPath( hash, full_size = False )
-        elif mime in HC.AUDIO: path = os.path.join( HC.STATIC_DIR, 'audio_resized.png' )
-        elif mime in HC.VIDEO: path = os.path.join( HC.STATIC_DIR, 'video_resized.png' )
-        elif mime == HC.APPLICATION_FLASH: path = os.path.join( HC.STATIC_DIR, 'flash_resized.png' )
-        elif mime == HC.APPLICATION_PDF: path = os.path.join( HC.STATIC_DIR, 'pdf_resized.png' )
-        else: path = os.path.join( HC.STATIC_DIR, 'hydrus_resized.png' )
+        elif mime in HC.AUDIO: path = os.path.join( HC.STATIC_DIR, 'audio.png' )
+        elif mime == HC.APPLICATION_PDF: path = os.path.join( HC.STATIC_DIR, 'pdf.png' )
+        else: path = os.path.join( HC.STATIC_DIR, 'hydrus.png' )
         
         response_context = HydrusServerResources.ResponseContext( 200, path = path )
         
@@ -259,7 +261,9 @@ class HydrusResourceCommandLocalFile( HydrusServerResources.HydrusResourceComman
         
         hash = request.hydrus_args[ 'hash' ]
         
-        path = ClientFiles.GetFilePath( hash )
+        client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+        
+        path = client_files_manager.GetFilePath( hash )
         
         response_context = HydrusServerResources.ResponseContext( 200, path = path )
         

include/ClientRendering.py

@@ -71,7 +71,9 @@ class RasterContainer( object ):
         hash = self._media.GetHash()
         mime = self._media.GetMime()
         
-        self._path = ClientFiles.GetFilePath( hash, mime )
+        client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+        
+        self._path = client_files_manager.GetFilePath( hash, mime )
         
         ( original_width, original_height ) = self._media.GetResolution()
         
@@ -174,7 +176,9 @@ class RasterContainerVideo( RasterContainer ):
         hash = self._media.GetHash()
         mime = self._media.GetMime()
         
-        path = ClientFiles.GetFilePath( hash, mime )
+        client_files_manager = HydrusGlobals.client_controller.GetClientFilesManager()
+        
+        path = client_files_manager.GetFilePath( hash, mime )
         
         duration = self._media.GetDuration()
         num_frames = self._media.GetNumFrames()

include/ClientSearch.py

@@ -82,14 +82,7 @@ def FilterPredicates( search_entry, predicates, service_key = None, expand_paren
     
     return matches
 
-def SortPredicates( predicates, collapse_siblings = False ):
-    
-    if collapse_siblings:
-        
-        siblings_manager = HydrusGlobals.client_controller.GetManager( 'tag_siblings' )
-        
-        predicates = siblings_manager.CollapsePredicates( predicates )
-        
+def SortPredicates( predicates ):
     
     def cmp_func( x, y ): return cmp( x.GetCount(), y.GetCount() )
     

include/HydrusConstants.py

@@ -29,6 +29,8 @@ CLIENT_THUMBNAILS_DIR = os.path.join( DB_DIR, 'client_thumbnails' )
 SERVER_THUMBNAILS_DIR = os.path.join( DB_DIR, 'server_thumbnails' )
 CLIENT_UPDATES_DIR = os.path.join( DB_DIR, 'client_updates' )
 SERVER_UPDATES_DIR = os.path.join( DB_DIR, 'server_updates' )
+HELP_DIR = os.path.join( BASE_DIR, 'help' )
+INCLUDE_DIR = os.path.join( BASE_DIR, 'include' )
 LOGS_DIR = os.path.join( BASE_DIR, 'logs' )
 STATIC_DIR = os.path.join( BASE_DIR, 'static' )
 
@@ -51,7 +53,7 @@ options = {}
 # Misc
 
 NETWORK_VERSION = 17
-SOFTWARE_VERSION = 183
+SOFTWARE_VERSION = 184
 
 UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )
 

include/HydrusData.py

@@ -741,6 +741,17 @@ def IsAlreadyRunning( instance ):
     
     return False
     
+def IterateHexPrefixes():
+    
+    hex_chars = '0123456789abcdef'
+    
+    for ( one, two ) in itertools.product( hex_chars, hex_chars ):
+        
+        prefix = one + two
+        
+        yield prefix
+        
+    
 def MergeKeyToListDicts( key_to_list_dicts ):
     
     result = collections.defaultdict( list )

include/HydrusGlobals.py

@@ -5,12 +5,9 @@ test_controller = None
 view_shutdown = False
 model_shutdown = False
 
-instance = None
-
 is_first_start = False
 is_db_updated = False
 
 db_profile_mode = False
 pubsub_profile_mode = False
-special_debug_mode = False
 server_busy = False

include/HydrusPaths.py

@@ -52,20 +52,48 @@ def CleanUpTempPath( os_file_handle, temp_path ):
     
 def ConvertAbsPathToPortablePath( abs_path ):
     
-    if abs_path == '': return None
-    
     try: return os.path.relpath( abs_path, HC.BASE_DIR )
     except: return abs_path
     
 def ConvertPortablePathToAbsPath( portable_path ):
     
-    if portable_path is None: return None
-    
     if os.path.isabs( portable_path ): abs_path = portable_path
     else: abs_path = os.path.normpath( os.path.join( HC.BASE_DIR, portable_path ) )
     
-    if os.path.exists( abs_path ): return abs_path
-    else: return None
+    return abs_path
+    
+def CopyAndMergeTree( source, dest ):
+    
+    if not os.path.exists( dest ):
+        
+        os.mkdir( dest )
+        
+    
+    for ( root, dirnames, filenames ) in os.walk( source ):
+        
+        dest_root = root.replace( source, dest )
+        
+        for dirname in dirnames:
+            
+            source_path = os.path.join( root, dirname )
+            dest_path = os.path.join( dest_root, dirname )
+            
+            if not os.path.exists( dest_path ):
+                
+                os.mkdir( dest_path )
+                
+            
+            shutil.copystat( source_path, dest_path )
+            
+        
+        for filename in filenames:
+            
+            source_path = os.path.join( root, filename )
+            dest_path = os.path.join( dest_root, filename )
+            
+            shutil.copy2( source_path, dest_path )
+            
+        
     
 def CopyFileLikeToFileLike( f_source, f_dest ):
     

include/HydrusServer.py

@@ -169,28 +169,6 @@ eris = '''<html><head><title>hydrus</title></head><body><pre>
                                      <font color="gray">MMMM</font>
 </pre></body></html>'''
 
-CLIENT_ROOT_MESSAGE = '''<html>
-    <head>
-        <title>hydrus client</title>
-    </head>
-    <body>
-        <p>This hydrus client uses software version ''' + str( HC.SOFTWARE_VERSION ) + ''' and network version ''' + str( HC.NETWORK_VERSION ) + '''.</p>
-        <p>It only serves requests from 127.0.0.1.</p>
-    </body>
-</html>'''
-
-ROOT_MESSAGE_BEGIN = '''<html>
-    <head>
-        <title>hydrus service</title>
-    </head>
-    <body>
-        <p>This hydrus service uses software version ''' + str( HC.SOFTWARE_VERSION ) + ''' and network version ''' + str( HC.NETWORK_VERSION ) + '''.</p>
-        <p>'''
-
-ROOT_MESSAGE_END = '''</p>
-    </body>
-</html>'''
-
 LOCAL_DOMAIN = HydrusServerResources.HydrusDomain( True )
 REMOTE_DOMAIN = HydrusServerResources.HydrusDomain( False )
 
@@ -225,15 +203,6 @@ class HydrusRequest( Request ):
         HydrusData.Print( message )
         
     
-class HydrusRequestRestricted( HydrusRequest ):
-    
-    def __init__( self, *args, **kwargs ):
-        
-        HydrusRequest.__init__( self, *args, **kwargs )
-        
-        self.hydrus_account = None
-        
-    
 class HydrusService( Site ):
     
     def __init__( self, service_key, service_type, message ):
@@ -258,79 +227,7 @@ class HydrusService( Site ):
         
         return root
         
-
-class HydrusServiceRestricted( HydrusService ):
     
-    def __init__( self, service_key, service_type, message ):
-        
-        HydrusService.__init__( self, service_key, service_type, message )
-        
-        self.requestFactory = HydrusRequestRestricted
-        
-    
-    def _InitRoot( self ):
-        
-        root = HydrusService._InitRoot( self )
-        
-        root.putChild( 'access_key', HydrusServerResources.HydrusResourceCommandAccessKey( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'access_key_verification', HydrusServerResources.HydrusResourceCommandAccessKeyVerification( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'session_key', HydrusServerResources.HydrusResourceCommandSessionKey( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        
-        root.putChild( 'account', HydrusServerResources.HydrusResourceCommandRestrictedAccount( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'account_info', HydrusServerResources.HydrusResourceCommandRestrictedAccountInfo( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'account_types', HydrusServerResources.HydrusResourceCommandRestrictedAccountTypes( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'registration_keys', HydrusServerResources.HydrusResourceCommandRestrictedRegistrationKeys( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'stats', HydrusServerResources.HydrusResourceCommandRestrictedStats( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        
-        return root
-        
-    
-class HydrusServiceAdmin( HydrusServiceRestricted ):
-    
-    def _InitRoot( self ):
-        
-        root = HydrusServiceRestricted._InitRoot( self )
-        
-        root.putChild( 'busy', HydrusServerResources.HydrusResourceBusyCheck() )
-        root.putChild( 'backup', HydrusServerResources.HydrusResourceCommandRestrictedBackup( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'init', HydrusServerResources.HydrusResourceCommandInit( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'services', HydrusServerResources.HydrusResourceCommandRestrictedServices( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'services_info', HydrusServerResources.HydrusResourceCommandRestrictedServicesInfo( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'shutdown', HydrusServerResources.HydrusResourceCommandShutdown( self._service_key, self._service_type, LOCAL_DOMAIN ) )
-        
-        return root
-        
-    
-class HydrusServiceRepository( HydrusServiceRestricted ):
-    
-    def _InitRoot( self ):
-        
-        root = HydrusServiceRestricted._InitRoot( self )
-        
-        root.putChild( 'news', HydrusServerResources.HydrusResourceCommandRestrictedNews( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'num_petitions', HydrusServerResources.HydrusResourceCommandRestrictedNumPetitions( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'petition', HydrusServerResources.HydrusResourceCommandRestrictedPetition( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'content_update_package', HydrusServerResources.HydrusResourceCommandRestrictedContentUpdate( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'immediate_content_update_package', HydrusServerResources.HydrusResourceCommandRestrictedImmediateContentUpdate( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'service_update_package', HydrusServerResources.HydrusResourceCommandRestrictedServiceUpdate( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        
-        return root
-        
-    
-class HydrusServiceRepositoryFile( HydrusServiceRepository ):
-    
-    def _InitRoot( self ):
-        
-        root = HydrusServiceRepository._InitRoot( self )
-        
-        root.putChild( 'file', HydrusServerResources.HydrusResourceCommandRestrictedRepositoryFile( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'ip', HydrusServerResources.HydrusResourceCommandRestrictedIP( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        root.putChild( 'thumbnail', HydrusServerResources.HydrusResourceCommandRestrictedRepositoryThumbnail( self._service_key, self._service_type, REMOTE_DOMAIN ) )
-        
-        return root
-        
-    
-class HydrusServiceRepositoryTag( HydrusServiceRepository ): pass
 '''
 class MessagingServiceFactory( ServerFactory ):
     

include/HydrusServerResources.py

@@ -1,13 +1,10 @@
-import Cookie
 import HydrusConstants as HC
 import HydrusExceptions
 import HydrusFileHandling
 import HydrusImageHandling
 import HydrusPaths
 import HydrusSerialisable
-import HydrusThreading
 import os
-import ServerFiles
 import time
 import traceback
 import yaml
@@ -88,7 +85,6 @@ def ParseFileArguments( path ):
     return args
     
 hydrus_favicon = FileResource( os.path.join( HC.STATIC_DIR, 'hydrus.ico' ), defaultType = 'image/x-icon' )
-local_booru_css = FileResource( os.path.join( HC.STATIC_DIR, 'local_booru_style.css' ), defaultType = 'text/css' )
 
 class HydrusDomain( object ):
     
@@ -102,23 +98,6 @@ class HydrusDomain( object ):
         if self._local_only and client_ip != '127.0.0.1': raise HydrusExceptions.ForbiddenException( 'Only local access allowed!' )
         
     
-class HydrusResourceBusyCheck( Resource ):
-    
-    def __init__( self ):
-        
-        Resource.__init__( self )
-        
-        self._server_version_string = HC.service_string_lookup[ HC.SERVER_ADMIN ] + '/' + str( HC.NETWORK_VERSION )
-        
-    
-    def render_GET( self, request ):
-        
-        request.setHeader( 'Server', self._server_version_string )
-        
-        if HydrusGlobals.server_busy: return '1'
-        else: return '0'
-        
-    
 class HydrusResourceWelcome( Resource ):
     
     def __init__( self, service_type, message ):
@@ -135,6 +114,8 @@ class HydrusResourceWelcome( Resource ):
     
     def render_GET( self, request ):
         
+        request.setResponseCode( 200 )
+        
         request.setHeader( 'Server', self._server_version_string )
         
         return self._body
@@ -583,524 +564,6 @@ class HydrusResourceCommand( Resource ):
         return NOT_DONE_YET
         
     
-class HydrusResourceCommandAccessKey( HydrusResourceCommand ):
-    
-    def _threadDoGETJob( self, request ):
-        
-        registration_key = self._parseAccessKey( request )
-        
-        access_key = HydrusGlobals.controller.Read( 'access_key', registration_key )
-        
-        body = yaml.safe_dump( { 'access_key' : access_key } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandShutdown( HydrusResourceCommand ):
-    
-    def _threadDoPOSTJob( self, request ):
-        
-        HydrusGlobals.controller.ShutdownFromServer()
-        
-        response_context = ResponseContext( 200 )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandAccessKeyVerification( HydrusResourceCommand ):
-    
-    def _threadDoGETJob( self, request ):
-        
-        access_key = self._parseAccessKey( request )
-        
-        verified = HydrusGlobals.controller.Read( 'verify_access_key', self._service_key, access_key )
-        
-        body = yaml.safe_dump( { 'verified' : verified } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandInit( HydrusResourceCommand ):
-    
-    def _threadDoGETJob( self, request ):
-        
-        access_key = HydrusGlobals.controller.Read( 'init' )
-        
-        body = yaml.safe_dump( { 'access_key' : access_key } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandSessionKey( HydrusResourceCommand ):
-    
-    def _threadDoGETJob( self, request ):
-        
-        access_key = self._parseAccessKey( request )
-        
-        session_manager = HydrusGlobals.controller.GetManager( 'restricted_services_sessions' )
-        
-        ( session_key, expires ) = session_manager.AddSession( self._service_key, access_key )
-        
-        now = HydrusData.GetNow()
-        
-        max_age = now - expires
-        
-        cookies = [ ( 'session_key', session_key.encode( 'hex' ), { 'max_age' : max_age, 'path' : '/' } ) ]
-        
-        response_context = ResponseContext( 200, cookies = cookies )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestricted( HydrusResourceCommand ):
-    
-    GET_PERMISSION = HC.GENERAL_ADMIN
-    POST_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _callbackCheckRestrictions( self, request ):
-        
-        self._checkServerBusy()
-        
-        self._checkUserAgent( request )
-        
-        self._domain.CheckValid( request.getClientIP() )
-        
-        self._checkSession( request )
-        
-        self._checkPermission( request )
-        
-        return request
-        
-    
-    def _checkPermission( self, request ):
-        
-        account = request.hydrus_account
-        
-        method = request.method
-        
-        permission = None
-        
-        if method == 'GET': permission = self.GET_PERMISSION
-        elif method == 'POST': permission = self.POST_PERMISSION
-        
-        if permission is not None: account.CheckPermission( permission )
-        
-        return request
-        
-    
-    def _checkSession( self, request ):
-        
-        if not request.requestHeaders.hasHeader( 'Cookie' ): raise HydrusExceptions.PermissionException( 'No cookies found!' )
-        
-        cookie_texts = request.requestHeaders.getRawHeaders( 'Cookie' )
-        
-        cookie_text = cookie_texts[0]
-        
-        try:
-            
-            cookies = Cookie.SimpleCookie( cookie_text )
-            
-            if 'session_key' not in cookies: session_key = None
-            else: session_key = cookies[ 'session_key' ].value.decode( 'hex' )
-            
-        except: raise Exception( 'Problem parsing cookies!' )
-        
-        session_manager = HydrusGlobals.controller.GetManager( 'restricted_services_sessions' )
-        
-        account = session_manager.GetAccount( self._service_key, session_key )
-        
-        request.hydrus_account = account
-        
-        return request
-        
-    
-    def _recordDataUsage( self, request ):
-        
-        path = request.path[1:] # /account -> account
-        
-        if request.method == 'GET': method = HC.GET
-        else: method = HC.POST
-        
-        if ( self._service_type, method, path ) in HC.BANDWIDTH_CONSUMING_REQUESTS:
-            
-            account = request.hydrus_account
-            
-            if account is not None:
-                
-                num_bytes = request.hydrus_request_data_usage
-                
-                account.RequestMade( num_bytes )
-                
-                HydrusGlobals.controller.pub( 'request_made', ( account.GetAccountKey(), num_bytes ) )
-                
-            
-        
-    
-class HydrusResourceCommandRestrictedAccount( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = None
-    POST_PERMISSION = HC.MANAGE_USERS
-    
-    def _threadDoGETJob( self, request ):
-        
-        account = request.hydrus_account
-        
-        body = yaml.safe_dump( { 'account' : account } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-    def _threadDoPOSTJob( self, request ):
-        
-        admin_account = request.hydrus_account
-        
-        admin_account_key = admin_account.GetAccountKey()
-        
-        action = request.hydrus_args[ 'action' ]
-        
-        subject_identifiers = request.hydrus_args[ 'subject_identifiers' ]
-        
-        subject_account_keys = { HydrusGlobals.controller.Read( 'account_key_from_identifier', self._service_key, subject_identifier ) for subject_identifier in subject_identifiers }
-        
-        kwargs = request.hydrus_args # for things like expires, title, and so on
-        
-        HydrusGlobals.controller.WriteSynchronous( 'account', self._service_key, admin_account_key, action, subject_account_keys, kwargs )
-        
-        session_manager = HydrusGlobals.controller.GetManager( 'restricted_services_sessions' )
-        
-        session_manager.RefreshAccounts( self._service_key, subject_account_keys )
-        
-        response_context = ResponseContext( 200 )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedAccountInfo( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _threadDoGETJob( self, request ):
-        
-        subject_identifier = request.hydrus_args[ 'subject_identifier' ]
-        
-        subject_account_key = HydrusGlobals.controller.Read( 'account_key_from_identifier', self._service_key, subject_identifier )
-        
-        account_info = HydrusGlobals.controller.Read( 'account_info', self._service_key, subject_account_key )
-        
-        body = yaml.safe_dump( { 'account_info' : account_info } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedAccountTypes( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GENERAL_ADMIN
-    POST_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _threadDoGETJob( self, request ):
-        
-        account_types = HydrusGlobals.controller.Read( 'account_types', self._service_key )
-        
-        body = yaml.safe_dump( { 'account_types' : account_types } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-    def _threadDoPOSTJob( self, request ):
-        
-        edit_log = request.hydrus_args[ 'edit_log' ]
-        
-        HydrusGlobals.controller.WriteSynchronous( 'account_types', self._service_key, edit_log )
-        
-        response_context = ResponseContext( 200 )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedBackup( HydrusResourceCommandRestricted ):
-    
-    POST_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _threadDoPOSTJob( self, request ):
-        
-        def do_it():
-            
-            HydrusGlobals.server_busy = True
-            
-            HydrusGlobals.controller.WriteSynchronous( 'backup' )
-            
-            HydrusGlobals.server_busy = False
-            
-        
-        HydrusGlobals.controller.CallToThread( do_it )
-        
-        response_context = ResponseContext( 200 )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedIP( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _threadDoGETJob( self, request ):
-        
-        hash = request.hydrus_args[ 'hash' ]
-        
-        ( ip, timestamp ) = HydrusGlobals.controller.Read( 'ip', self._service_key, hash )
-        
-        body = yaml.safe_dump( { 'ip' : ip, 'timestamp' : timestamp } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedNews( HydrusResourceCommandRestricted ):
-    
-    POST_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _threadDoPOSTJob( self, request ):
-        
-        news = request.hydrus_args[ 'news' ]
-        
-        HydrusGlobals.controller.WriteSynchronous( 'news', self._service_key, news )
-        
-        response_context = ResponseContext( 200 )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedNumPetitions( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.RESOLVE_PETITIONS
-    
-    def _threadDoGETJob( self, request ):
-        
-        num_petitions = HydrusGlobals.controller.Read( 'num_petitions', self._service_key )
-        
-        body = yaml.safe_dump( { 'num_petitions' : num_petitions } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedPetition( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.RESOLVE_PETITIONS
-    
-    def _threadDoGETJob( self, request ):
-        
-        petition = HydrusGlobals.controller.Read( 'petition', self._service_key )
-        
-        body = petition.DumpToNetworkString()
-        
-        response_context = ResponseContext( 200, mime = HC.APPLICATION_JSON, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedRegistrationKeys( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _threadDoGETJob( self, request ):
-        
-        num = request.hydrus_args[ 'num' ]
-        title = request.hydrus_args[ 'title' ]
-        
-        if 'lifetime' in request.hydrus_args: lifetime = request.hydrus_args[ 'lifetime' ]
-        else: lifetime = None
-        
-        registration_keys = HydrusGlobals.controller.Read( 'registration_keys', self._service_key, num, title, lifetime )
-        
-        body = yaml.safe_dump( { 'registration_keys' : registration_keys } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedRepositoryFile( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GET_DATA
-    POST_PERMISSION = HC.POST_DATA
-    
-    def _threadDoGETJob( self, request ):
-        
-        hash = request.hydrus_args[ 'hash' ]
-        
-        # don't I need to check that we aren't stealing the file from another service?
-        
-        path = ServerFiles.GetPath( 'file', hash )
-        
-        response_context = ResponseContext( 200, path = path )
-        
-        return response_context
-        
-    
-    def _threadDoPOSTJob( self, request ):
-        
-        account = request.hydrus_account
-        
-        account_key = account.GetAccountKey()
-        
-        file_dict = request.hydrus_args
-        
-        file_dict[ 'ip' ] = request.getClientIP()
-        
-        HydrusGlobals.controller.WriteSynchronous( 'file', self._service_key, account_key, file_dict )
-        
-        response_context = ResponseContext( 200 )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedRepositoryThumbnail( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GET_DATA
-    
-    def _threadDoGETJob( self, request ):
-        
-        hash = request.hydrus_args[ 'hash' ]
-        
-        # don't I need to check that we aren't stealing the file from another service?
-        
-        path = ServerFiles.GetPath( 'thumbnail', hash )
-        
-        response_context = ResponseContext( 200, path = path )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedServices( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GENERAL_ADMIN
-    POST_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _threadDoPOSTJob( self, request ):
-        
-        account = request.hydrus_account
-        
-        account_key = account.GetAccountKey()
-        
-        edit_log = request.hydrus_args[ 'edit_log' ]
-        
-        service_keys_to_access_keys = HydrusGlobals.controller.WriteSynchronous( 'services', account_key, edit_log )
-        
-        body = yaml.safe_dump( { 'service_keys_to_access_keys' : service_keys_to_access_keys } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedServicesInfo( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GENERAL_ADMIN
-    POST_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _threadDoGETJob( self, request ):
-        
-        services_info = HydrusGlobals.controller.Read( 'services_info' )
-        
-        body = yaml.safe_dump( { 'services_info' : services_info } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedStats( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GENERAL_ADMIN
-    
-    def _threadDoGETJob( self, request ):
-        
-        stats = HydrusGlobals.controller.Read( 'stats', self._service_key )
-        
-        body = yaml.safe_dump( { 'stats' : stats } )
-        
-        response_context = ResponseContext( 200, body = body )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedContentUpdate( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GET_DATA
-    POST_PERMISSION = HC.POST_DATA
-    
-    def _threadDoGETJob( self, request ):
-        
-        begin = request.hydrus_args[ 'begin' ]
-        subindex = request.hydrus_args[ 'subindex' ]
-        
-        path = ServerFiles.GetContentUpdatePackagePath( self._service_key, begin, subindex )
-        
-        response_context = ResponseContext( 200, path = path, is_json = True )
-        
-        return response_context
-        
-    
-    def _threadDoPOSTJob( self, request ):
-        
-        account = request.hydrus_account
-        
-        account_key = account.GetAccountKey()
-        
-        update = request.hydrus_args[ 'update' ]
-        
-        HydrusGlobals.controller.WriteSynchronous( 'update', self._service_key, account_key, update )
-        
-        response_context = ResponseContext( 200 )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedImmediateContentUpdate( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.RESOLVE_PETITIONS
-    
-    def _threadDoGETJob( self, request ):
-        
-        content_update = HydrusGlobals.controller.Read( 'immediate_content_update', self._service_key )
-        
-        network_string = content_update.DumpToNetworkString()
-        
-        response_context = ResponseContext( 200, mime = HC.APPLICATION_JSON, body = network_string )
-        
-        return response_context
-        
-    
-class HydrusResourceCommandRestrictedServiceUpdate( HydrusResourceCommandRestricted ):
-    
-    GET_PERMISSION = HC.GET_DATA
-    
-    def _threadDoGETJob( self, request ):
-        
-        begin = request.hydrus_args[ 'begin' ]
-        
-        path = ServerFiles.GetServiceUpdatePackagePath( self._service_key, begin )
-        
-        response_context = ResponseContext( 200, path = path, is_json = True )
-        
-        return response_context
-        
-    
 class ResponseContext( object ):
     
     def __init__( self, status_code, mime = HC.APPLICATION_YAML, body = None, path = None, is_json = False, cookies = None ):

include/ServerController.py

@@ -5,12 +5,12 @@ import HydrusData
 import HydrusExceptions
 import HydrusGlobals
 import HydrusNetworking
-import HydrusServer
 import HydrusSessions
 import HydrusThreading
 import os
 import ServerDaemons
 import ServerDB
+import ServerServer
 import sys
 import time
 import traceback
@@ -227,9 +227,9 @@ class Controller( HydrusController.HydrusController ):
                         
                         message = options[ 'message' ]
                         
-                        if service_type == HC.SERVER_ADMIN: service_object = HydrusServer.HydrusServiceAdmin( service_key, service_type, message )
-                        elif service_type == HC.FILE_REPOSITORY: service_object = HydrusServer.HydrusServiceRepositoryFile( service_key, service_type, message )
-                        elif service_type == HC.TAG_REPOSITORY: service_object = HydrusServer.HydrusServiceRepositoryTag( service_key, service_type, message )
+                        if service_type == HC.SERVER_ADMIN: service_object = ServerServer.HydrusServiceAdmin( service_key, service_type, message )
+                        elif service_type == HC.FILE_REPOSITORY: service_object = ServerServer.HydrusServiceRepositoryFile( service_key, service_type, message )
+                        elif service_type == HC.TAG_REPOSITORY: service_object = ServerServer.HydrusServiceRepositoryTag( service_key, service_type, message )
                         elif service_type == HC.MESSAGE_DEPOT: return
                         
                         self._services[ service_key ] = reactor.listenTCP( port, service_object )

include/ServerDB.py

@@ -541,13 +541,11 @@ class DB( HydrusDB.HydrusDB ):
         
         dirs = ( HC.SERVER_FILES_DIR, HC.SERVER_THUMBNAILS_DIR )
         
-        hex_chars = '0123456789abcdef'
-        
         for dir in dirs:
             
-            for ( one, two ) in itertools.product( hex_chars, hex_chars ):
+            for prefix in HydrusData.IterateHexPrefixes():
                 
-                new_dir = os.path.join( dir, one + two )
+                new_dir = os.path.join( dir, prefix )
                 
                 if not os.path.exists( new_dir ):
                     
@@ -1767,7 +1765,7 @@ class DB( HydrusDB.HydrusDB ):
         os.mkdir( backup_path )
         
         HydrusData.Print( 'backing up: copying db file' )
-        shutil.copy( self._db_path, os.path.join( backup_path, self.DB_NAME + '.db' ) )
+        shutil.copy2( self._db_path, os.path.join( backup_path, self.DB_NAME + '.db' ) )
         
         HydrusData.Print( 'backing up: copying files' )
         shutil.copytree( HC.SERVER_FILES_DIR, os.path.join( backup_path, 'server_files' ) )
@@ -2350,13 +2348,11 @@ class DB( HydrusDB.HydrusDB ):
             
             dirs = ( HC.SERVER_FILES_DIR, HC.SERVER_THUMBNAILS_DIR )
             
-            hex_chars = '0123456789abcdef'
-            
             for dir in dirs:
                 
-                for ( one, two ) in itertools.product( hex_chars, hex_chars ):
+                for prefix in HydrusData.IterateHexPrefixes():
                     
-                    new_dir = os.path.join( dir, one + two )
+                    new_dir = os.path.join( dir, prefix )
                     
                     if not os.path.exists( new_dir ):
                         

include/ServerServer.py

@@ -0,0 +1,84 @@
+import HydrusServer
+import ServerServerResources
+
+class HydrusRequestRestricted( HydrusServer.HydrusRequest ):
+    
+    def __init__( self, *args, **kwargs ):
+        
+        HydrusServer.HydrusRequest.__init__( self, *args, **kwargs )
+        
+        self.hydrus_account = None
+        
+    
+class HydrusServiceRestricted( HydrusServer.HydrusService ):
+    
+    def __init__( self, service_key, service_type, message ):
+        
+        HydrusServer.HydrusService.__init__( self, service_key, service_type, message )
+        
+        self.requestFactory = HydrusRequestRestricted
+        
+    
+    def _InitRoot( self ):
+        
+        root = HydrusServer.HydrusService._InitRoot( self )
+        
+        root.putChild( 'access_key', ServerServerResources.HydrusResourceCommandAccessKey( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'access_key_verification', ServerServerResources.HydrusResourceCommandAccessKeyVerification( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'session_key', ServerServerResources.HydrusResourceCommandSessionKey( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        
+        root.putChild( 'account', ServerServerResources.HydrusResourceCommandRestrictedAccount( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'account_info', ServerServerResources.HydrusResourceCommandRestrictedAccountInfo( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'account_types', ServerServerResources.HydrusResourceCommandRestrictedAccountTypes( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'registration_keys', ServerServerResources.HydrusResourceCommandRestrictedRegistrationKeys( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'stats', ServerServerResources.HydrusResourceCommandRestrictedStats( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        
+        return root
+        
+    
+class HydrusServiceAdmin( HydrusServiceRestricted ):
+    
+    def _InitRoot( self ):
+        
+        root = HydrusServiceRestricted._InitRoot( self )
+        
+        root.putChild( 'busy', ServerServerResources.HydrusResourceBusyCheck() )
+        root.putChild( 'backup', ServerServerResources.HydrusResourceCommandRestrictedBackup( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'init', ServerServerResources.HydrusResourceCommandInit( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'services', ServerServerResources.HydrusResourceCommandRestrictedServices( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'services_info', ServerServerResources.HydrusResourceCommandRestrictedServicesInfo( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'shutdown', ServerServerResources.HydrusResourceCommandShutdown( self._service_key, self._service_type, HydrusServer.LOCAL_DOMAIN ) )
+        
+        return root
+        
+    
+class HydrusServiceRepository( HydrusServiceRestricted ):
+    
+    def _InitRoot( self ):
+        
+        root = HydrusServiceRestricted._InitRoot( self )
+        
+        root.putChild( 'news', ServerServerResources.HydrusResourceCommandRestrictedNews( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'num_petitions', ServerServerResources.HydrusResourceCommandRestrictedNumPetitions( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'petition', ServerServerResources.HydrusResourceCommandRestrictedPetition( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'content_update_package', ServerServerResources.HydrusResourceCommandRestrictedContentUpdate( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'immediate_content_update_package', ServerServerResources.HydrusResourceCommandRestrictedImmediateContentUpdate( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'service_update_package', ServerServerResources.HydrusResourceCommandRestrictedServiceUpdate( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        
+        return root
+        
+    
+class HydrusServiceRepositoryFile( HydrusServiceRepository ):
+    
+    def _InitRoot( self ):
+        
+        root = HydrusServiceRepository._InitRoot( self )
+        
+        root.putChild( 'file', ServerServerResources.HydrusResourceCommandRestrictedRepositoryFile( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'ip', ServerServerResources.HydrusResourceCommandRestrictedIP( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        root.putChild( 'thumbnail', ServerServerResources.HydrusResourceCommandRestrictedRepositoryThumbnail( self._service_key, self._service_type, HydrusServer.REMOTE_DOMAIN ) )
+        
+        return root
+        
+    
+class HydrusServiceRepositoryTag( HydrusServiceRepository ): pass

include/ServerServerResources.py

@@ -0,0 +1,546 @@
+import Cookie
+import HydrusConstants as HC
+import HydrusData
+import HydrusExceptions
+import HydrusGlobals
+import HydrusServerResources
+import ServerFiles
+import yaml
+
+class HydrusResourceBusyCheck( HydrusServerResources.Resource ):
+    
+    def __init__( self ):
+        
+        HydrusServerResources.Resource.__init__( self )
+        
+        self._server_version_string = HC.service_string_lookup[ HC.SERVER_ADMIN ] + '/' + str( HC.NETWORK_VERSION )
+        
+    
+    def render_GET( self, request ):
+        
+        request.setResponseCode( 200 )
+        
+        request.setHeader( 'Server', self._server_version_string )
+        
+        if HydrusGlobals.server_busy: return '1'
+        else: return '0'
+        
+    
+class HydrusResourceCommandAccessKey( HydrusServerResources.HydrusResourceCommand ):
+    
+    def _threadDoGETJob( self, request ):
+        
+        registration_key = self._parseAccessKey( request )
+        
+        access_key = HydrusGlobals.controller.Read( 'access_key', registration_key )
+        
+        body = yaml.safe_dump( { 'access_key' : access_key } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandShutdown( HydrusServerResources.HydrusResourceCommand ):
+    
+    def _threadDoPOSTJob( self, request ):
+        
+        HydrusGlobals.controller.ShutdownFromServer()
+        
+        response_context = HydrusServerResources.ResponseContext( 200 )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandAccessKeyVerification( HydrusServerResources.HydrusResourceCommand ):
+    
+    def _threadDoGETJob( self, request ):
+        
+        access_key = self._parseAccessKey( request )
+        
+        verified = HydrusGlobals.controller.Read( 'verify_access_key', self._service_key, access_key )
+        
+        body = yaml.safe_dump( { 'verified' : verified } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandInit( HydrusServerResources.HydrusResourceCommand ):
+    
+    def _threadDoGETJob( self, request ):
+        
+        access_key = HydrusGlobals.controller.Read( 'init' )
+        
+        body = yaml.safe_dump( { 'access_key' : access_key } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandSessionKey( HydrusServerResources.HydrusResourceCommand ):
+    
+    def _threadDoGETJob( self, request ):
+        
+        access_key = self._parseAccessKey( request )
+        
+        session_manager = HydrusGlobals.controller.GetManager( 'restricted_services_sessions' )
+        
+        ( session_key, expires ) = session_manager.AddSession( self._service_key, access_key )
+        
+        now = HydrusData.GetNow()
+        
+        max_age = now - expires
+        
+        cookies = [ ( 'session_key', session_key.encode( 'hex' ), { 'max_age' : max_age, 'path' : '/' } ) ]
+        
+        response_context = HydrusServerResources.ResponseContext( 200, cookies = cookies )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestricted( HydrusServerResources.HydrusResourceCommand ):
+    
+    GET_PERMISSION = HC.GENERAL_ADMIN
+    POST_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _callbackCheckRestrictions( self, request ):
+        
+        self._checkServerBusy()
+        
+        self._checkUserAgent( request )
+        
+        self._domain.CheckValid( request.getClientIP() )
+        
+        self._checkSession( request )
+        
+        self._checkPermission( request )
+        
+        return request
+        
+    
+    def _checkPermission( self, request ):
+        
+        account = request.hydrus_account
+        
+        method = request.method
+        
+        permission = None
+        
+        if method == 'GET': permission = self.GET_PERMISSION
+        elif method == 'POST': permission = self.POST_PERMISSION
+        
+        if permission is not None: account.CheckPermission( permission )
+        
+        return request
+        
+    
+    def _checkSession( self, request ):
+        
+        if not request.requestHeaders.hasHeader( 'Cookie' ): raise HydrusExceptions.PermissionException( 'No cookies found!' )
+        
+        cookie_texts = request.requestHeaders.getRawHeaders( 'Cookie' )
+        
+        cookie_text = cookie_texts[0]
+        
+        try:
+            
+            cookies = Cookie.SimpleCookie( cookie_text )
+            
+            if 'session_key' not in cookies: session_key = None
+            else: session_key = cookies[ 'session_key' ].value.decode( 'hex' )
+            
+        except: raise Exception( 'Problem parsing cookies!' )
+        
+        session_manager = HydrusGlobals.controller.GetManager( 'restricted_services_sessions' )
+        
+        account = session_manager.GetAccount( self._service_key, session_key )
+        
+        request.hydrus_account = account
+        
+        return request
+        
+    
+    def _recordDataUsage( self, request ):
+        
+        path = request.path[1:] # /account -> account
+        
+        if request.method == 'GET': method = HC.GET
+        else: method = HC.POST
+        
+        if ( self._service_type, method, path ) in HC.BANDWIDTH_CONSUMING_REQUESTS:
+            
+            account = request.hydrus_account
+            
+            if account is not None:
+                
+                num_bytes = request.hydrus_request_data_usage
+                
+                account.RequestMade( num_bytes )
+                
+                HydrusGlobals.controller.pub( 'request_made', ( account.GetAccountKey(), num_bytes ) )
+                
+            
+        
+    
+class HydrusResourceCommandRestrictedAccount( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = None
+    POST_PERMISSION = HC.MANAGE_USERS
+    
+    def _threadDoGETJob( self, request ):
+        
+        account = request.hydrus_account
+        
+        body = yaml.safe_dump( { 'account' : account } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+    def _threadDoPOSTJob( self, request ):
+        
+        admin_account = request.hydrus_account
+        
+        admin_account_key = admin_account.GetAccountKey()
+        
+        action = request.hydrus_args[ 'action' ]
+        
+        subject_identifiers = request.hydrus_args[ 'subject_identifiers' ]
+        
+        subject_account_keys = { HydrusGlobals.controller.Read( 'account_key_from_identifier', self._service_key, subject_identifier ) for subject_identifier in subject_identifiers }
+        
+        kwargs = request.hydrus_args # for things like expires, title, and so on
+        
+        HydrusGlobals.controller.WriteSynchronous( 'account', self._service_key, admin_account_key, action, subject_account_keys, kwargs )
+        
+        session_manager = HydrusGlobals.controller.GetManager( 'restricted_services_sessions' )
+        
+        session_manager.RefreshAccounts( self._service_key, subject_account_keys )
+        
+        response_context = HydrusServerResources.ResponseContext( 200 )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedAccountInfo( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _threadDoGETJob( self, request ):
+        
+        subject_identifier = request.hydrus_args[ 'subject_identifier' ]
+        
+        subject_account_key = HydrusGlobals.controller.Read( 'account_key_from_identifier', self._service_key, subject_identifier )
+        
+        account_info = HydrusGlobals.controller.Read( 'account_info', self._service_key, subject_account_key )
+        
+        body = yaml.safe_dump( { 'account_info' : account_info } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedAccountTypes( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GENERAL_ADMIN
+    POST_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _threadDoGETJob( self, request ):
+        
+        account_types = HydrusGlobals.controller.Read( 'account_types', self._service_key )
+        
+        body = yaml.safe_dump( { 'account_types' : account_types } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+    def _threadDoPOSTJob( self, request ):
+        
+        edit_log = request.hydrus_args[ 'edit_log' ]
+        
+        HydrusGlobals.controller.WriteSynchronous( 'account_types', self._service_key, edit_log )
+        
+        response_context = HydrusServerResources.ResponseContext( 200 )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedBackup( HydrusResourceCommandRestricted ):
+    
+    POST_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _threadDoPOSTJob( self, request ):
+        
+        def do_it():
+            
+            HydrusGlobals.server_busy = True
+            
+            HydrusGlobals.controller.WriteSynchronous( 'backup' )
+            
+            HydrusGlobals.server_busy = False
+            
+        
+        HydrusGlobals.controller.CallToThread( do_it )
+        
+        response_context = HydrusServerResources.ResponseContext( 200 )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedIP( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _threadDoGETJob( self, request ):
+        
+        hash = request.hydrus_args[ 'hash' ]
+        
+        ( ip, timestamp ) = HydrusGlobals.controller.Read( 'ip', self._service_key, hash )
+        
+        body = yaml.safe_dump( { 'ip' : ip, 'timestamp' : timestamp } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedNews( HydrusResourceCommandRestricted ):
+    
+    POST_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _threadDoPOSTJob( self, request ):
+        
+        news = request.hydrus_args[ 'news' ]
+        
+        HydrusGlobals.controller.WriteSynchronous( 'news', self._service_key, news )
+        
+        response_context = HydrusServerResources.ResponseContext( 200 )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedNumPetitions( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.RESOLVE_PETITIONS
+    
+    def _threadDoGETJob( self, request ):
+        
+        num_petitions = HydrusGlobals.controller.Read( 'num_petitions', self._service_key )
+        
+        body = yaml.safe_dump( { 'num_petitions' : num_petitions } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedPetition( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.RESOLVE_PETITIONS
+    
+    def _threadDoGETJob( self, request ):
+        
+        petition = HydrusGlobals.controller.Read( 'petition', self._service_key )
+        
+        body = petition.DumpToNetworkString()
+        
+        response_context = HydrusServerResources.ResponseContext( 200, mime = HC.APPLICATION_JSON, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedRegistrationKeys( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _threadDoGETJob( self, request ):
+        
+        num = request.hydrus_args[ 'num' ]
+        title = request.hydrus_args[ 'title' ]
+        
+        if 'lifetime' in request.hydrus_args: lifetime = request.hydrus_args[ 'lifetime' ]
+        else: lifetime = None
+        
+        registration_keys = HydrusGlobals.controller.Read( 'registration_keys', self._service_key, num, title, lifetime )
+        
+        body = yaml.safe_dump( { 'registration_keys' : registration_keys } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedRepositoryFile( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GET_DATA
+    POST_PERMISSION = HC.POST_DATA
+    
+    def _threadDoGETJob( self, request ):
+        
+        hash = request.hydrus_args[ 'hash' ]
+        
+        # don't I need to check that we aren't stealing the file from another service?
+        
+        path = ServerFiles.GetPath( 'file', hash )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, path = path )
+        
+        return response_context
+        
+    
+    def _threadDoPOSTJob( self, request ):
+        
+        account = request.hydrus_account
+        
+        account_key = account.GetAccountKey()
+        
+        file_dict = request.hydrus_args
+        
+        file_dict[ 'ip' ] = request.getClientIP()
+        
+        HydrusGlobals.controller.WriteSynchronous( 'file', self._service_key, account_key, file_dict )
+        
+        response_context = HydrusServerResources.ResponseContext( 200 )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedRepositoryThumbnail( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GET_DATA
+    
+    def _threadDoGETJob( self, request ):
+        
+        hash = request.hydrus_args[ 'hash' ]
+        
+        # don't I need to check that we aren't stealing the file from another service?
+        
+        path = ServerFiles.GetPath( 'thumbnail', hash )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, path = path )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedServices( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GENERAL_ADMIN
+    POST_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _threadDoPOSTJob( self, request ):
+        
+        account = request.hydrus_account
+        
+        account_key = account.GetAccountKey()
+        
+        edit_log = request.hydrus_args[ 'edit_log' ]
+        
+        service_keys_to_access_keys = HydrusGlobals.controller.WriteSynchronous( 'services', account_key, edit_log )
+        
+        body = yaml.safe_dump( { 'service_keys_to_access_keys' : service_keys_to_access_keys } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedServicesInfo( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GENERAL_ADMIN
+    POST_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _threadDoGETJob( self, request ):
+        
+        services_info = HydrusGlobals.controller.Read( 'services_info' )
+        
+        body = yaml.safe_dump( { 'services_info' : services_info } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedStats( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GENERAL_ADMIN
+    
+    def _threadDoGETJob( self, request ):
+        
+        stats = HydrusGlobals.controller.Read( 'stats', self._service_key )
+        
+        body = yaml.safe_dump( { 'stats' : stats } )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, body = body )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedContentUpdate( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GET_DATA
+    POST_PERMISSION = HC.POST_DATA
+    
+    def _threadDoGETJob( self, request ):
+        
+        begin = request.hydrus_args[ 'begin' ]
+        subindex = request.hydrus_args[ 'subindex' ]
+        
+        path = ServerFiles.GetContentUpdatePackagePath( self._service_key, begin, subindex )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, path = path, is_json = True )
+        
+        return response_context
+        
+    
+    def _threadDoPOSTJob( self, request ):
+        
+        account = request.hydrus_account
+        
+        account_key = account.GetAccountKey()
+        
+        update = request.hydrus_args[ 'update' ]
+        
+        HydrusGlobals.controller.WriteSynchronous( 'update', self._service_key, account_key, update )
+        
+        response_context = HydrusServerResources.ResponseContext( 200 )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedImmediateContentUpdate( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.RESOLVE_PETITIONS
+    
+    def _threadDoGETJob( self, request ):
+        
+        content_update = HydrusGlobals.controller.Read( 'immediate_content_update', self._service_key )
+        
+        network_string = content_update.DumpToNetworkString()
+        
+        response_context = HydrusServerResources.ResponseContext( 200, mime = HC.APPLICATION_JSON, body = network_string )
+        
+        return response_context
+        
+    
+class HydrusResourceCommandRestrictedServiceUpdate( HydrusResourceCommandRestricted ):
+    
+    GET_PERMISSION = HC.GET_DATA
+    
+    def _threadDoGETJob( self, request ):
+        
+        begin = request.hydrus_args[ 'begin' ]
+        
+        path = ServerFiles.GetServiceUpdatePackagePath( self._service_key, begin )
+        
+        response_context = HydrusServerResources.ResponseContext( 200, path = path, is_json = True )
+        
+        return response_context
+        
+    

include/TestDB.py

@@ -47,15 +47,6 @@ class TestClientDB( unittest.TestCase ):
     @classmethod
     def setUpClass( self ):
         
-        self._old_db_dir = HC.DB_DIR
-        self._old_client_files_dir = HC.CLIENT_FILES_DIR
-        self._old_client_thumbnails_dir = HC.CLIENT_THUMBNAILS_DIR
-        
-        HC.DB_DIR = tempfile.mkdtemp()
-        
-        HC.CLIENT_FILES_DIR = os.path.join( HC.DB_DIR, 'client_files' )
-        HC.CLIENT_THUMBNAILS_DIR = os.path.join( HC.DB_DIR, 'client_thumbnails' )
-        
         self._db = ClientDB.DB( HydrusGlobals.test_controller )
         
         threading.Thread( target = self._db.MainLoop, name = 'Database Main Loop' ).start()
@@ -68,12 +59,6 @@ class TestClientDB( unittest.TestCase ):
         
         while not self._db.LoopIsFinished(): time.sleep( 0.1 )
         
-        shutil.rmtree( HC.DB_DIR )
-        
-        HC.DB_DIR = self._old_db_dir
-        HC.CLIENT_FILES_DIR = self._old_client_files_dir
-        HC.CLIENT_THUMBNAILS_DIR = self._old_client_thumbnails_dir
-        
     
     def test_4chan_pass( self ):
         
@@ -738,16 +723,14 @@ class TestClientDB( unittest.TestCase ):
         self.assertTrue( os.path.exists( HC.CLIENT_FILES_DIR ) )
         
         self.assertTrue( os.path.exists( HC.CLIENT_THUMBNAILS_DIR ) )
-    
-        hex_chars = '0123456789abcdef'
         
-        for ( one, two ) in itertools.product( hex_chars, hex_chars ):
+        for prefix in HydrusData.IterateHexPrefixes():
             
-            dir = os.path.join( HC.CLIENT_FILES_DIR, one + two )
+            dir = os.path.join( HC.CLIENT_FILES_DIR, prefix )
             
             self.assertTrue( os.path.exists( dir ) )
             
-            dir = os.path.join( HC.CLIENT_THUMBNAILS_DIR, one + two )
+            dir = os.path.join( HC.CLIENT_THUMBNAILS_DIR, prefix )
             
             self.assertTrue( os.path.exists( dir ) )
             

include/TestHydrusServer.py

@@ -12,6 +12,7 @@ import HydrusSerialisable
 import itertools
 import os
 import ServerFiles
+import ServerServer
 import shutil
 import stat
 import TestConstants
@@ -60,11 +61,11 @@ class TestServer( unittest.TestCase ):
         
         def TWISTEDSetup():
             
-            reactor.listenTCP( HC.DEFAULT_SERVER_ADMIN_PORT, HydrusServer.HydrusServiceAdmin( self._admin_service.GetServiceKey(), HC.SERVER_ADMIN, 'hello' ) )
+            reactor.listenTCP( HC.DEFAULT_SERVER_ADMIN_PORT, ServerServer.HydrusServiceAdmin( self._admin_service.GetServiceKey(), HC.SERVER_ADMIN, 'hello' ) )
             reactor.listenTCP( HC.DEFAULT_LOCAL_FILE_PORT, ClientLocalServer.HydrusServiceLocal( CC.LOCAL_FILE_SERVICE_KEY, HC.LOCAL_FILE, 'hello' ) )
             reactor.listenTCP( HC.DEFAULT_LOCAL_BOORU_PORT, ClientLocalServer.HydrusServiceBooru( CC.LOCAL_BOORU_SERVICE_KEY, HC.LOCAL_BOORU, 'hello' ) )
-            reactor.listenTCP( HC.DEFAULT_SERVICE_PORT, HydrusServer.HydrusServiceRepositoryFile( self._file_service.GetServiceKey(), HC.FILE_REPOSITORY, 'hello' ) )
-            reactor.listenTCP( HC.DEFAULT_SERVICE_PORT + 1, HydrusServer.HydrusServiceRepositoryTag( self._tag_service.GetServiceKey(), HC.TAG_REPOSITORY, 'hello' ) )
+            reactor.listenTCP( HC.DEFAULT_SERVICE_PORT, ServerServer.HydrusServiceRepositoryFile( self._file_service.GetServiceKey(), HC.FILE_REPOSITORY, 'hello' ) )
+            reactor.listenTCP( HC.DEFAULT_SERVICE_PORT + 1, ServerServer.HydrusServiceRepositoryTag( self._tag_service.GetServiceKey(), HC.TAG_REPOSITORY, 'hello' ) )
             
         
         reactor.callFromThread( TWISTEDSetup )
@@ -115,7 +116,7 @@ class TestServer( unittest.TestCase ):
         
         #
         
-        path = ClientFiles.GetExpectedFilePath( self._file_hash, HC.IMAGE_JPEG )
+        path = ClientFiles.GetExpectedFilePath( HC.CLIENT_FILES_DIR, self._file_hash, HC.IMAGE_JPEG )
         
         with open( path, 'wb' ) as f: f.write( 'file' )
         
@@ -237,7 +238,7 @@ class TestServer( unittest.TestCase ):
         share_key = HydrusData.GenerateKey()
         hashes = [ HydrusData.GenerateKey() for i in range( 5 ) ]
         
-        with open( ClientFiles.GetExpectedFilePath( hashes[0], HC.IMAGE_JPEG ), 'wb' ) as f: f.write( 'file' )
+        with open( ClientFiles.GetExpectedFilePath( HC.CLIENT_FILES_DIR, hashes[0], HC.IMAGE_JPEG ), 'wb' ) as f: f.write( 'file' )
         with open( ClientFiles.GetExpectedThumbnailPath( hashes[0], False ), 'wb' ) as f: f.write( 'thumbnail' )
         
         local_booru_manager = HydrusGlobals.test_controller.GetManager( 'local_booru' )

server.py

@@ -28,8 +28,6 @@ try:
     from include import HydrusLogger
     import traceback
     
-    HydrusGlobals.instance = HC.HYDRUS_SERVER
-    
     action = ServerController.GetStartingAction()
     
     if action == 'help':
@@ -47,8 +45,6 @@ try:
         
         with HydrusLogger.HydrusLogger( 'server.log' ) as logger:
             
-            error_occured = False
-            
             try:
                 
                 if action in ( 'stop', 'restart' ):
@@ -69,14 +65,12 @@ try:
                 
             except HydrusExceptions.PermissionException as e:
                 
-                error_occured = True
                 error = str( e )
                 
                 HydrusData.Print( error )
                 
             except:
                 
-                error_occured = True
                 error = traceback.format_exc()
                 
                 HydrusData.Print( 'Hydrus server failed' )

test.py

@@ -30,7 +30,9 @@ from include import TestHydrusTags
 import collections
 import os
 import random
+import shutil
 import sys
+import tempfile
 import threading
 import time
 import unittest
@@ -40,8 +42,6 @@ from include import ClientCaches
 from include import ClientData
 from include import HydrusData
 
-HydrusGlobals.instance = HC.HYDRUS_TEST
-
 only_run = None
 
 class Controller( object ):
@@ -54,6 +54,15 @@ class Controller( object ):
         HydrusGlobals.test_controller = self
         self._pubsub = HydrusPubSub.HydrusPubSub( self )
         
+        self._old_db_dir = HC.DB_DIR
+        self._old_client_files_dir = HC.CLIENT_FILES_DIR
+        self._old_client_thumbnails_dir = HC.CLIENT_THUMBNAILS_DIR
+        
+        HC.DB_DIR = tempfile.mkdtemp()
+        
+        HC.CLIENT_FILES_DIR = os.path.join( HC.DB_DIR, 'client_files' )
+        HC.CLIENT_THUMBNAILS_DIR = os.path.join( HC.DB_DIR, 'client_thumbnails' )
+        
         self._new_options = ClientData.ClientOptions()
         
         def show_text( text ): pass
@@ -78,6 +87,10 @@ class Controller( object ):
         services.append( ClientData.Service( CC.LOCAL_TAG_SERVICE_KEY, HC.LOCAL_TAG, CC.LOCAL_TAG_SERVICE_KEY, {} ) )
         self._reads[ 'services' ] = services
         
+        client_files_locations = { prefix : HC.CLIENT_FILES_DIR for prefix in HydrusData.IterateHexPrefixes() }
+        
+        self._reads[ 'client_files_locations' ] = client_files_locations
+        
         self._reads[ 'sessions' ] = []
         self._reads[ 'tag_parents' ] = {}
         self._reads[ 'tag_siblings' ] = {}
@@ -90,6 +103,7 @@ class Controller( object ):
         self._managers = {}
         
         self._services_manager = ClientCaches.ServicesManager( self )
+        self._client_files_manager = ClientCaches.ClientFilesManager( self )
         
         self._managers[ 'hydrus_sessions' ] = ClientCaches.HydrusSessionManager( self )
         self._managers[ 'tag_censorship' ] = ClientCaches.TagCensorshipManager( self )
@@ -130,6 +144,11 @@ class Controller( object ):
     
     def DoHTTP( self, *args, **kwargs ): return self._http.Request( *args, **kwargs )
     
+    def GetClientFilesManager( self ):
+        
+        return self._client_files_manager
+        
+    
     def GetHTTP( self ): return self._http
     
     def GetNewOptions( self ):
@@ -200,6 +219,15 @@ class Controller( object ):
     
     def SetWebCookies( self, name, value ): self._cookies[ name ] = value
     
+    def TidyUp( self ):
+        
+        shutil.rmtree( HC.DB_DIR )
+        
+        HC.DB_DIR = self._old_db_dir
+        HC.CLIENT_FILES_DIR = self._old_client_files_dir
+        HC.CLIENT_THUMBNAILS_DIR = self._old_client_thumbnails_dir
+        
+    
     def ViewIsShutdown( self ):
         
         return HydrusGlobals.view_shutdown
@@ -268,6 +296,8 @@ if __name__ == '__main__':
         
         controller.pubimmediate( 'wake_daemons' )
         
+        controller.TidyUp()
+        
         reactor.callFromThread( reactor.stop )
         
         raw_input()