changelog
-
+
version 262
+ - added apng support! +
- sessions will now append from the default insertion index (e.g. right of current page) +
- sessions will now load in the correct order if the default insertion index is to the left of the current page! +
- fixed an issue with custom 'favorites' gelbooru parsers +
- manage parents/siblings dialogs now load their larger datasets on a thread, after initialising +
- sped up how some default thumbnails are generated on client boot +
- sped up how resized thumbnails are generated +
- refactored ffmpeg video parsing to be simpler and easier to maintain +
- improved some ffmpeg parsing to be much faster +
- ffmpeg can now parse video that reports no duration +
- fixed some more bad video framerate parsing +
- the 4chan/8chan thread watcher will now always generate https urls (even if you enter an http thread url) +
- pixiv will now produce https urls +
- the known url file status system will now check both http and https versions of an url it is given +
- refactored some core elements to simplify common controller requests +
- started on some database migration help and ui--should be finished next week +
- flushed out network engine a little +
- more refactoring and simplification of new network engine object coupling +
- flushed out bandwidth manager +
- misc logic tweaks in bandwidth management +
- wrote a network session manager +
- wrote a new class for identifying all future 'network contexts' +
- employed NetworkContext everywhere in the new engine, massively simplifying several things +
- wrote the first batch of tests for my new network engine and its network jobs +
- wrote tests for the bandwidth rules object, including covering the new short time delta support +
- improved reliability and speed of bandwidth unit tests +
- wrote tests for the new domain-based bandwidth manager +
- misc improvements +
- more misc improvements +
version 261
- wrote a new manage dialog for urls diff --git a/help/database_migration.html b/help/database_migration.html new file mode 100644 index 00000000..ad214fc0 --- /dev/null +++ b/help/database_migration.html @@ -0,0 +1,56 @@ + + +
-
+ the software installation
+
This is the part that comes with the installer or extract release, with the executable and dlls and a handful of resource folders. It doesn't store any of your settings--it just knows how to present a database as a nice application. If you just run the client executable straight, it looks in its 'db' subdirectory for a database, and if one is not found, it creates a new one. If it sees a database running at a lower version than itself, it will update the database before booting it.
+It doesn't really matter where you put this. An SSD will load it marginally quicker the first time, but you probably won't notice. If you run it without command-line parameters, it will try to write to its own directory (to create the initial database), so if you mean to run it like that, it should not be in a protected place like Program Files.
+
+ -
+ the actual database
+
The client stores all its preferences and currently open tabs and knowledge about files--like file size and resolution, tags, ratings, inbox status, and so on and so on--in a handful of SQLite database files, defaulting to install_dir/db. Depending on the size of your client, these might total 1MB in size or be as much as 10GB.
+In order to perform a search or to fetch or process tags, the client has to interact with these files in many small bursts, which means it is best if these files are on a drive with low latency. An SSD is ideal, but a regularly-defragged HDD with a reasonable amount of free space also works well.
+
+ -
+ your media files
+
All of your jpegs and webms and so on (and their thumbnails) are stored in a single complicated directory that is by default at install_dir/db/client_files. All the files are named by their hash and stored in efficient hash-based subdirectories. In general, it is not navigable by humans, but it works very well for the fast access from a giant pool of files the client needs to do to manage your media.
+Thumbnails tend to be fetched dozens at a time, so it is, again, ideal if they are stored on an SSD. Your regular media files--which on many clients total hundreds of GB--are usually fetched one at a time for human consumption and do not benefit from the expensive low-latency of an SSD. They are best stored on a cheap HDD, and, if desired, also work well when stored across a network file system.
+
+ - client -d="D:\media\my_hydrus_database" +
- --or-- +
- client --db_dir="G:\misc documents\New Folder (3)\DO NOT ENTER" +
-
+
+
+
+
\ No newline at end of file
diff --git a/help/db_migration_shortcut.png b/help/db_migration_shortcut.png
new file mode 100644
index 00000000..80a365e7
Binary files /dev/null and b/help/db_migration_shortcut.png differ
diff --git a/include/ClientCaches.py b/include/ClientCaches.py
index f4725238..8ab8977c 100644
--- a/include/ClientCaches.py
+++ b/include/ClientCaches.py
@@ -306,7 +306,7 @@ class ClientFilesManager( object ):
try:
- thumbnail_resized = HydrusFileHandling.GenerateThumbnail( full_size_path, thumbnail_dimensions )
+ thumbnail_resized = HydrusFileHandling.GenerateThumbnailFromStaticImage( full_size_path, thumbnail_dimensions )
except:
@@ -321,7 +321,7 @@ class ClientFilesManager( object ):
self._GenerateFullSizeThumbnail( hash )
- thumbnail_resized = HydrusFileHandling.GenerateThumbnail( full_size_path, thumbnail_dimensions )
+ thumbnail_resized = HydrusFileHandling.GenerateThumbnailFromStaticImage( full_size_path, thumbnail_dimensions )
resized_path = self._GenerateExpectedResizedThumbnailPath( hash )
@@ -1457,7 +1457,7 @@ class LocalBooruCache( object ):
def _RefreshShares( self ):
- self._local_booru_service = self._controller.GetServicesManager().GetService( CC.LOCAL_BOORU_SERVICE_KEY )
+ self._local_booru_service = self._controller.services_manager.GetService( CC.LOCAL_BOORU_SERVICE_KEY )
self._keys_to_infos = {}
@@ -1572,7 +1572,7 @@ class HydrusSessionManager( object ):
# session key expired or not found
- service = self._controller.GetServicesManager().GetService( service_key )
+ service = self._controller.services_manager.GetService( service_key )
( response_gumpf, cookies ) = service.Request( HC.GET, 'session_key', return_cookies = True )
@@ -1740,7 +1740,6 @@ class ThumbnailCache( object ):
cache_size = options[ 'thumbnail_cache_size' ]
self._data_cache = DataCache( self._controller, cache_size )
- self._client_files_manager = self._controller.GetClientFilesManager()
self._lock = threading.Lock()
@@ -1783,11 +1782,11 @@ class ThumbnailCache( object ):
if full_size:
- path = self._client_files_manager.GetFullSizeThumbnailPath( hash )
+ path = self._controller.client_files_manager.GetFullSizeThumbnailPath( hash )
else:
- path = self._client_files_manager.GetResizedThumbnailPath( hash )
+ path = self._controller.client_files_manager.GetResizedThumbnailPath( hash )
except HydrusExceptions.FileMissingException as e:
@@ -1803,11 +1802,11 @@ class ThumbnailCache( object ):
if full_size:
- path = self._client_files_manager.GetFullSizeThumbnailPath( hash )
+ path = self._controller.client_files_manager.GetFullSizeThumbnailPath( hash )
else:
- path = self._client_files_manager.GetResizedThumbnailPath( hash )
+ path = self._controller.client_files_manager.GetResizedThumbnailPath( hash )
except HydrusExceptions.FileMissingException:
@@ -1826,7 +1825,7 @@ class ThumbnailCache( object ):
try:
- self._client_files_manager.RegenerateResizedThumbnail( hash )
+ self._controller.client_files_manager.RegenerateResizedThumbnail( hash )
try:
@@ -1861,7 +1860,7 @@ class ThumbnailCache( object ):
if too_large or ( too_small and not small_original_image ):
- self._client_files_manager.RegenerateResizedThumbnail( hash )
+ self._controller.client_files_manager.RegenerateResizedThumbnail( hash )
hydrus_bitmap = ClientRendering.GenerateHydrusBitmap( path )
@@ -1906,7 +1905,7 @@ class ThumbnailCache( object ):
options = self._controller.GetOptions()
- thumbnail = HydrusFileHandling.GenerateThumbnail( path, options[ 'thumbnail_dimensions' ] )
+ thumbnail = HydrusFileHandling.GenerateThumbnailFromStaticImage( path, options[ 'thumbnail_dimensions' ] )
with open( temp_path, 'wb' ) as f:
@@ -3260,7 +3259,7 @@ class WebSessionManagerClient( object ):
form_fields[ 'password' ] = password
form_fields[ 'captcha' ] = ''
form_fields[ 'g_recaptcha_response' ] = ''
- form_fields[ 'return_to' ] = 'http://www.pixiv.net'
+ form_fields[ 'return_to' ] = 'https://www.pixiv.net'
form_fields[ 'lang' ] = 'en'
form_fields[ 'post_key' ] = post_key
form_fields[ 'source' ] = 'pc'
diff --git a/include/ClientConstants.py b/include/ClientConstants.py
index 6c4656ba..6fccd24b 100755
--- a/include/ClientConstants.py
+++ b/include/ClientConstants.py
@@ -195,6 +195,7 @@ media_viewer_capabilities = {}
media_viewer_capabilities[ HC.IMAGE_JPEG ] = static_full_support
media_viewer_capabilities[ HC.IMAGE_PNG ] = static_full_support
+media_viewer_capabilities[ HC.IMAGE_APNG ] = animated_full_support
media_viewer_capabilities[ HC.IMAGE_GIF ] = animated_full_support
if HC.PLATFORM_WINDOWS:
@@ -240,10 +241,16 @@ NEW_PAGE_GOES_FAR_RIGHT = 3
new_page_goes_string_lookup = {}
new_page_goes_string_lookup[ NEW_PAGE_GOES_FAR_LEFT ] = 'go far left'
-new_page_goes_string_lookup[ NEW_PAGE_GOES_LEFT_OF_CURRENT ] = 'go left of current page'
-new_page_goes_string_lookup[ NEW_PAGE_GOES_RIGHT_OF_CURRENT ] = 'go right of current page'
+new_page_goes_string_lookup[ NEW_PAGE_GOES_LEFT_OF_CURRENT ] = 'go left of current page tab'
+new_page_goes_string_lookup[ NEW_PAGE_GOES_RIGHT_OF_CURRENT ] = 'go right of current page tab'
new_page_goes_string_lookup[ NEW_PAGE_GOES_FAR_RIGHT ] = 'go far right'
+NETWORK_CONTEXT_GLOBAL = 0
+NETWORK_CONTEXT_HYDRUS = 1
+NETWORK_CONTEXT_DOMAIN = 1
+NETWORK_CONTEXT_DOWNLOADER = 1
+NETWORK_CONTEXT_SUBSCRIPTION = 1
+
SHORTCUT_MODIFIER_CTRL = 0
SHORTCUT_MODIFIER_ALT = 1
SHORTCUT_MODIFIER_SHIFT = 2
diff --git a/include/ClientController.py b/include/ClientController.py
index 5483b89a..01e4b895 100755
--- a/include/ClientController.py
+++ b/include/ClientController.py
@@ -57,6 +57,9 @@ class Controller( HydrusController.HydrusController ):
self._previously_idle = False
self._idle_started = None
+ self.client_files_manager = None
+ self.services_manager = None
+
def _InitDB( self ):
@@ -391,13 +394,13 @@ class Controller( HydrusController.HydrusController ):
stop_time = HydrusData.GetNow() + ( self._options[ 'idle_shutdown_max_minutes' ] * 60 )
- self._client_files_manager.Rebalance( partial = False, stop_time = stop_time )
+ self.client_files_manager.Rebalance( partial = False, stop_time = stop_time )
self.MaintainDB( stop_time = stop_time )
if not self._options[ 'pause_repo_sync' ]:
- services = self.GetServicesManager().GetServices( HC.REPOSITORIES )
+ services = self.services_manager.GetServices( HC.REPOSITORIES )
for service in services:
@@ -488,11 +491,6 @@ class Controller( HydrusController.HydrusController ):
raise NotImplementedError()
- def GetClientFilesManager( self ):
-
- return self._client_files_manager
-
-
def GetClientSessionManager( self ):
return self._client_session_manager
@@ -503,7 +501,10 @@ class Controller( HydrusController.HydrusController ):
return self._shortcuts_manager.GetCommand( shortcut_names, shortcut )
- def GetGUI( self ): return self._gui
+ def GetGUI( self ):
+
+ return self._gui
+
def GetOptions( self ):
@@ -515,11 +516,6 @@ class Controller( HydrusController.HydrusController ):
return self._new_options
- def GetServicesManager( self ):
-
- return self._services_manager
-
-
def GoodTimeToDoForegroundWork( self ):
if self._gui:
@@ -534,9 +530,9 @@ class Controller( HydrusController.HydrusController ):
def InitClientFilesManager( self ):
- self._client_files_manager = ClientCaches.ClientFilesManager( self )
+ self.client_files_manager = ClientCaches.ClientFilesManager( self )
- missing_locations = self._client_files_manager.GetMissing()
+ missing_locations = self.client_files_manager.GetMissing()
while len( missing_locations ) > 0:
@@ -548,9 +544,9 @@ class Controller( HydrusController.HydrusController ):
if dlg.ShowModal() == wx.ID_OK:
- self._client_files_manager = ClientCaches.ClientFilesManager( self )
+ self.client_files_manager = ClientCaches.ClientFilesManager( self )
- missing_locations = self._client_files_manager.GetMissing()
+ missing_locations = self.client_files_manager.GetMissing()
else:
@@ -568,7 +564,7 @@ class Controller( HydrusController.HydrusController ):
HydrusController.HydrusController.InitModel( self )
- self._services_manager = ClientCaches.ServicesManager( self )
+ self.services_manager = ClientCaches.ServicesManager( self )
self._options = self.Read( 'options' )
self._new_options = self.Read( 'serialisable', HydrusSerialisable.SERIALISABLE_TYPE_CLIENT_OPTIONS )
@@ -769,7 +765,7 @@ class Controller( HydrusController.HydrusController ):
self.pub( 'splash_set_status_text', 'fattening service info' )
- services = self.GetServicesManager().GetServices()
+ services = self.services_manager.GetServices()
for service in services:
@@ -870,7 +866,7 @@ class Controller( HydrusController.HydrusController ):
def RefreshServices( self ):
- self._services_manager.RefreshServices()
+ self.services_manager.RefreshServices()
def ResetIdleTimer( self ):
@@ -885,7 +881,7 @@ class Controller( HydrusController.HydrusController ):
def RestartBooru( self ):
- service = self.GetServicesManager().GetService( CC.LOCAL_BOORU_SERVICE_KEY )
+ service = self.services_manager.GetService( CC.LOCAL_BOORU_SERVICE_KEY )
port = service.GetPort()
@@ -1028,7 +1024,7 @@ class Controller( HydrusController.HydrusController ):
with HG.dirty_object_lock:
- dirty_services = [ service for service in self._services_manager.GetServices() if service.IsDirty() ]
+ dirty_services = [ service for service in self.services_manager.GetServices() if service.IsDirty() ]
if len( dirty_services ) > 0:
@@ -1043,7 +1039,7 @@ class Controller( HydrusController.HydrusController ):
self.WriteSynchronous( 'update_services', services )
- self._services_manager.RefreshServices()
+ self.services_manager.RefreshServices()
@@ -1120,7 +1116,7 @@ class Controller( HydrusController.HydrusController ):
return True
- services = self.GetServicesManager().GetServices( HC.REPOSITORIES )
+ services = self.services_manager.GetServices( HC.REPOSITORIES )
for service in services:
diff --git a/include/ClientDB.py b/include/ClientDB.py
index c98cfb7e..f5ec843d 100755
--- a/include/ClientDB.py
+++ b/include/ClientDB.py
@@ -1423,7 +1423,7 @@ class DB( HydrusDB.HydrusDB ):
hash_ids = [ hash_id for ( hash_id, ) in self._c.execute( 'SELECT hash_id FROM shape_maintenance_phash_regen;' ) ]
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
total_done_previously = total_num_hash_ids_in_cache - len( hash_ids )
@@ -2470,7 +2470,7 @@ class DB( HydrusDB.HydrusDB ):
missing_count = 0
deletee_hash_ids = []
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
for ( i, ( hash_id, mime ) ) in enumerate( info ):
@@ -2933,7 +2933,7 @@ class DB( HydrusDB.HydrusDB ):
deletable_file_hash_ids = hash_ids.difference( potentially_pending_upload_hash_ids )
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
if len( deletable_file_hash_ids ) > 0:
@@ -5438,7 +5438,7 @@ class DB( HydrusDB.HydrusDB ):
needed_hashes = []
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
for hash_id in needed_hash_ids:
@@ -6060,18 +6060,30 @@ class DB( HydrusDB.HydrusDB ):
def _GetURLStatus( self, url ):
- result = self._c.execute( 'SELECT hash_id FROM urls WHERE url = ?;', ( url, ) ).fetchone()
+ search_urls = [ url ]
- if result is None:
+ if url.startswith( 'http://' ):
- return ( CC.STATUS_NEW, None )
+ search_urls.append( 'https://' + url[7:] )
- else:
+ elif url.startswith( 'https://' ):
- ( hash_id, ) = result
+ search_urls.append( 'http://' + url[8:] )
- return self._GetHashIdStatus( hash_id )
+
+ for search_url in search_urls:
+ result = self._c.execute( 'SELECT hash_id FROM urls WHERE url = ?;', ( search_url, ) ).fetchone()
+
+ if result is not None:
+
+ ( hash_id, ) = result
+
+ return self._GetHashIdStatus( hash_id )
+
+
+
+ return ( CC.STATUS_NEW, None )
def _GetWebSessions( self ):
@@ -6217,7 +6229,7 @@ class DB( HydrusDB.HydrusDB ):
timestamp = HydrusData.GetNow()
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
if mime in HC.MIMES_WITH_THUMBNAILS:
@@ -6314,7 +6326,7 @@ class DB( HydrusDB.HydrusDB ):
num_frames = None
num_words = None
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
# lockless because this db call is made by the locked client files manager
client_files_manager.LocklessAddFileFromString( update_hash, mime, update_network_string )
@@ -7610,7 +7622,7 @@ class DB( HydrusDB.HydrusDB ):
num_updates_done = 0
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
select_statement = 'SELECT hash_id FROM files_info WHERE mime = ' + str( HC.APPLICATION_HYDRUS_UPDATE_DEFINITIONS ) + ' AND hash_id IN %s;'
@@ -7830,7 +7842,7 @@ class DB( HydrusDB.HydrusDB ):
self._controller.pub( 'message', job_key )
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
num_to_do = len( hashes )
diff --git a/include/ClientDaemons.py b/include/ClientDaemons.py
index 2b454d06..8330d45f 100644
--- a/include/ClientDaemons.py
+++ b/include/ClientDaemons.py
@@ -62,7 +62,7 @@ def DAEMONDownloadFiles( controller ):
if num_downloads > 0:
- client_files_manager = controller.GetClientFilesManager()
+ client_files_manager = controller.client_files_manager
successful_hashes = set()
@@ -89,7 +89,7 @@ def DAEMONDownloadFiles( controller ):
try:
- service = controller.GetServicesManager().GetService( service_key )
+ service = controller.services_manager.GetService( service_key )
except:
@@ -233,7 +233,7 @@ def DAEMONMaintainTrash( controller ):
def DAEMONRebalanceClientFiles( controller ):
- controller.GetClientFilesManager().Rebalance()
+ controller.client_files_manager.Rebalance()
def DAEMONSaveDirtyObjects( controller ):
@@ -241,7 +241,7 @@ def DAEMONSaveDirtyObjects( controller ):
def DAEMONSynchroniseAccounts( controller ):
- services = controller.GetServicesManager().GetServices( HC.RESTRICTED_SERVICES )
+ services = controller.services_manager.GetServices( HC.RESTRICTED_SERVICES )
for service in services:
@@ -254,7 +254,7 @@ def DAEMONSynchroniseRepositories( controller ):
if not options[ 'pause_repo_sync' ]:
- services = controller.GetServicesManager().GetServices( HC.REPOSITORIES )
+ services = controller.services_manager.GetServices( HC.REPOSITORIES )
for service in services:
@@ -306,7 +306,7 @@ def DAEMONUPnP( controller ):
return # This IGD probably doesn't support UPnP, so don't spam the user with errors they can't fix!
- services = controller.GetServicesManager().GetServices( ( HC.LOCAL_BOORU, ) )
+ services = controller.services_manager.GetServices( ( HC.LOCAL_BOORU, ) )
for service in services:
diff --git a/include/ClientData.py b/include/ClientData.py
index 8764fba4..2f240fc7 100644
--- a/include/ClientData.py
+++ b/include/ClientData.py
@@ -126,7 +126,7 @@ def ConvertServiceKeysToContentUpdatesToPrettyString( service_keys_to_content_up
if len( content_updates ) > 0:
- name = HG.client_controller.GetServicesManager().GetName( service_key )
+ name = HG.client_controller.services_manager.GetName( service_key )
locations.add( name )
@@ -347,7 +347,7 @@ def GetSortChoices( add_namespaces_and_ratings = True ):
sort_choices.extend( HC.options[ 'sort_by' ] )
- service_keys = HG.client_controller.GetServicesManager().GetServiceKeys( ( HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ) )
+ service_keys = HG.client_controller.services_manager.GetServiceKeys( ( HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ) )
for service_key in service_keys:
@@ -679,7 +679,7 @@ class ApplicationCommand( HydrusSerialisable.SerialisableBase ):
components.append( '"' + HydrusData.ToUnicode( value ) + '"' )
components.append( 'for' )
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
if services_manager.ServiceExists( service_key ):
@@ -899,6 +899,7 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
self._dictionary[ 'media_view' ][ HC.IMAGE_JPEG ] = ( CC.MEDIA_VIEWER_ACTION_SHOW_AS_NORMAL, CC.MEDIA_VIEWER_ACTION_SHOW_AS_NORMAL, image_zoom_info )
self._dictionary[ 'media_view' ][ HC.IMAGE_PNG ] = ( CC.MEDIA_VIEWER_ACTION_SHOW_AS_NORMAL, CC.MEDIA_VIEWER_ACTION_SHOW_AS_NORMAL, image_zoom_info )
+ self._dictionary[ 'media_view' ][ HC.IMAGE_APNG ] = ( CC.MEDIA_VIEWER_ACTION_SHOW_AS_NORMAL, CC.MEDIA_VIEWER_ACTION_SHOW_AS_NORMAL, gif_zoom_info )
self._dictionary[ 'media_view' ][ HC.IMAGE_GIF ] = ( CC.MEDIA_VIEWER_ACTION_SHOW_AS_NORMAL, CC.MEDIA_VIEWER_ACTION_SHOW_AS_NORMAL, gif_zoom_info )
if HC.PLATFORM_WINDOWS:
@@ -1566,7 +1567,7 @@ class DuplicateActionOptions( HydrusSerialisable.SerialisableBase ):
if HG.client_controller.IsBooted():
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
self._tag_service_actions = [ ( service_key, action, tag_censor ) for ( service_key, action, tag_censor ) in self._tag_service_actions if services_manager.ServiceExists( service_key ) and services_manager.GetServiceType( service_key ) in ( HC.LOCAL_TAG, HC.TAG_REPOSITORY ) ]
self._rating_service_actions = [ ( service_key, action ) for ( service_key, action ) in self._rating_service_actions if services_manager.ServiceExists( service_key ) and services_manager.GetServiceType( service_key ) in ( HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ) ]
@@ -1664,7 +1665,7 @@ class DuplicateActionOptions( HydrusSerialisable.SerialisableBase ):
service_keys_to_content_updates = {}
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
for ( service_key, action, tag_censor ) in self._tag_service_actions:
@@ -1992,7 +1993,7 @@ class ImportTagOptions( HydrusSerialisable.SerialisableBase ):
if HG.client_controller.IsBooted():
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
test_func = services_manager.ServiceExists
@@ -2263,7 +2264,7 @@ class Shortcuts( HydrusSerialisable.SerialisableBaseNamed ):
# this never stored mouse actions, so skip
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
for ( modifier, key, ( serialisable_service_key, data ) ) in serialisable_keyboard_actions:
diff --git a/include/ClientDownloading.py b/include/ClientDownloading.py
index 44e7948b..58cef173 100644
--- a/include/ClientDownloading.py
+++ b/include/ClientDownloading.py
@@ -137,7 +137,7 @@ def GetImageboardFileURL( thread_url, filename, ext ):
if is_4chan:
- return 'http://i.4cdn.org/' + board + '/' + filename + ext
+ return 'https://i.4cdn.org/' + board + '/' + filename + ext
elif is_8chan:
@@ -151,7 +151,7 @@ def GetImageboardFileURL( thread_url, filename, ext ):
try:
- html_url = 'http://8ch.net/' + board + '/res/' + thread_id + '.html'
+ html_url = 'https://8ch.net/' + board + '/res/' + thread_id + '.html'
response = ClientNetworking.RequestsGet( html_url )
@@ -182,7 +182,7 @@ def GetImageboardFileURL( thread_url, filename, ext ):
media_host = _8CHAN_BOARDS_TO_MEDIA_HOSTS[ board ]
- return 'http://' + media_host + '/' + board + '/src/' + filename + ext
+ return 'https://' + media_host + '/' + board + '/src/' + filename + ext
@@ -194,18 +194,18 @@ def GetImageboardThreadJSONURL( thread_url ):
is_8chan = '8ch.net' in host
# 4chan
- # http://a.4cdn.org/asp/thread/382059.json
+ # https://a.4cdn.org/asp/thread/382059.json
# 8chan
- # http://8ch.net/v/res/406061.json
+ # https://8ch.net/v/res/406061.json
if is_4chan:
- return 'http://a.4cdn.org/' + board + '/thread/' + thread_id + '.json'
+ return 'https://a.4cdn.org/' + board + '/thread/' + thread_id + '.json'
elif is_8chan:
- return 'http://8ch.net/' + board + '/res/' + thread_id + '.json'
+ return 'https://8ch.net/' + board + '/res/' + thread_id + '.json'
def GetSoup( html ):
@@ -245,7 +245,7 @@ def THREADDownloadURL( job_key, url, url_string ):
job_key.SetVariable( 'popup_text_1', 'importing' )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
( result, hash ) = client_files_manager.ImportFile( temp_path )
@@ -336,7 +336,7 @@ def THREADDownloadURLs( job_key, urls, title ):
job_key.SetVariable( 'popup_text_2', 'importing' )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
( result, hash ) = client_files_manager.ImportFile( temp_path )
@@ -879,27 +879,34 @@ class GalleryBooru( Gallery ):
# https://gelbooru.com/redirect.php?s=Ly9nZWxib29ydS5jb20vaW5kZXgucGhwP3BhZ2U9cG9zdCZzPXZpZXcmaWQ9MzY5NDEyMg==
- try:
+ if 'redirect.php' in bad_url:
- encoded_location = bad_url.split( '?s=' )[1]
+ try:
+
+ encoded_location = bad_url.split( '?s=' )[1]
+
+ location = encoded_location.decode( 'base64' )
+
+ url = urlparse.urljoin( bad_url, location )
+
+ urls.append( url )
+
+ except Exception as e:
+
+ HydrusData.ShowText( 'gelbooru parsing problem!' )
+ HydrusData.ShowException( e )
+
+ url = ClientNetworking.RequestsGetRedirectURL( bad_url, session )
+
+ urls.append( url )
+
+ time.sleep( 0.5 )
+
- location = encoded_location.decode( 'base64' )
-
- url = urlparse.urljoin( bad_url, location )
+ else:
urls.append( url )
- except Exception as e:
-
- HydrusData.ShowText( 'gelbooru parsing problem!' )
- HydrusData.ShowException( e )
-
- url = ClientNetworking.RequestsGetRedirectURL( bad_url, session )
-
- urls.append( url )
-
- time.sleep( 0.5 )
-
@@ -1752,7 +1759,7 @@ class GalleryPixivArtistID( GalleryPixiv ):
artist_id = query
- gallery_url = 'http://www.pixiv.net/member_illust.php?type=illust&id=' + str( artist_id )
+ gallery_url = 'https://www.pixiv.net/member_illust.php?type=illust&id=' + str( artist_id )
return gallery_url + '&p=' + str( page_index + 1 )
@@ -1763,7 +1770,7 @@ class GalleryPixivTag( GalleryPixiv ):
tag = query
- gallery_url = 'http://www.pixiv.net/search.php?word=' + urllib.quote( HydrusData.ToByteString( tag ), '' ) + '&s_mode=s_tag_full&order=date_d'
+ gallery_url = 'https://www.pixiv.net/search.php?word=' + urllib.quote( HydrusData.ToByteString( tag ), '' ) + '&s_mode=s_tag_full&order=date_d'
return gallery_url + '&p=' + str( page_index + 1 )
diff --git a/include/ClientExporting.py b/include/ClientExporting.py
index 06139d7d..53b78afe 100644
--- a/include/ClientExporting.py
+++ b/include/ClientExporting.py
@@ -287,7 +287,7 @@ class ExportFolder( HydrusSerialisable.SerialisableBaseNamed ):
sync_filenames = set()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
num_copied = 0
diff --git a/include/ClientGUI.py b/include/ClientGUI.py
index 58e4dd59..d20abdf1 100755
--- a/include/ClientGUI.py
+++ b/include/ClientGUI.py
@@ -242,7 +242,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
subject_account_key = dlg.GetValue().decode( 'hex' )
- service = self._controller.GetServicesManager().GetService( service_key )
+ service = self._controller.services_manager.GetService( service_key )
response = service.Request( HC.GET, 'account_info', { 'subject_account_key' : subject_account_key } )
@@ -280,7 +280,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _AppendGUISession( self, name ):
- def do_it( session ):
+ def do_it( session, starting_index ):
try:
@@ -291,6 +291,8 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
wx.CallAfter( self._notebook.Disable )
+ forced_insertion_index = starting_index
+
for ( page_name, management_controller, initial_hashes ) in session.IteratePages():
try:
@@ -315,7 +317,9 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
initial_media_results = []
- wx.CallAfter( self._NewPage, page_name, management_controller, initial_media_results = initial_media_results )
+ wx.CallAfter( self._NewPage, page_name, management_controller, initial_media_results = initial_media_results, forced_insertion_index = forced_insertion_index )
+
+ forced_insertion_index += 1
except Exception as e:
@@ -358,7 +362,9 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
return
- self._controller.CallToThread( do_it, session )
+ starting_index = self._GetDefaultPageInsertionIndex()
+
+ self._controller.CallToThread( do_it, session, starting_index )
def _AutoRepoSetup( self ):
@@ -395,7 +401,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
file_repo.SetCredentials( credentials )
- all_services = self._controller.GetServicesManager().GetServices()
+ all_services = self._controller.services_manager.GetServices()
all_services.append( tag_repo )
all_services.append( file_repo )
@@ -540,13 +546,13 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
#
- all_services = list( self._controller.GetServicesManager().GetServices() )
+ all_services = list( self._controller.services_manager.GetServices() )
all_services.append( admin_service )
self._controller.SetServices( all_services )
- admin_service = self._controller.GetServicesManager().GetService( admin_service_key ) # let's refresh it
+ admin_service = self._controller.services_manager.GetService( admin_service_key ) # let's refresh it
HydrusData.ShowText( 'Admin service initialised.' )
@@ -607,7 +613,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
started = HydrusData.GetNow()
- service = self._controller.GetServicesManager().GetService( service_key )
+ service = self._controller.services_manager.GetService( service_key )
service.Request( HC.POST, 'backup' )
@@ -660,7 +666,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _CheckFileIntegrity( self ):
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
message = 'This will go through all the files the database thinks it has and check that they actually exist. Any files that are missing will be deleted from the internal record.'
message += os.linesep * 2
@@ -727,7 +733,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
text = 'What would you like to do with the orphaned files? Note that all orphaned thumbnails will be deleted.'
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
with ClientGUIDialogs.DialogYesNo( self, text, title = 'Choose what do to with the orphans.', yes_label = 'move them somewhere', no_label = 'delete them' ) as dlg_2:
@@ -928,7 +934,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _DeletePending( self, service_key ):
- service = self._controller.GetServicesManager().GetService( service_key )
+ service = self._controller.services_manager.GetService( service_key )
with ClientGUIDialogs.DialogYesNo( self, 'Are you sure you want to delete the pending data for ' + service.GetName() + '?' ) as dlg:
@@ -972,7 +978,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
hash = dlg.GetValue().decode( 'hex' )
- service = self._controller.GetServicesManager().GetService( service_key )
+ service = self._controller.services_manager.GetService( service_key )
with wx.BusyCursor(): response = service.Request( HC.GET, 'ip', { 'hash' : hash } )
@@ -1156,7 +1162,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
search_menu = wx.Menu()
- services = self._controller.GetServicesManager().GetServices()
+ services = self._controller.services_manager.GetServices()
petition_permissions = [ ( content_type, HC.PERMISSION_ACTION_OVERRULE ) for content_type in HC.REPOSITORY_CONTENT_TYPES ]
@@ -1303,7 +1309,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
for ( service_key, info ) in nums_pending.items():
- service = self._controller.GetServicesManager().GetService( service_key )
+ service = self._controller.services_manager.GetService( service_key )
service_type = service.GetServiceType()
name = service.GetName()
@@ -1369,8 +1375,8 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def services():
- tag_services = self._controller.GetServicesManager().GetServices( ( HC.TAG_REPOSITORY, ) )
- file_services = self._controller.GetServicesManager().GetServices( ( HC.FILE_REPOSITORY, ) )
+ tag_services = self._controller.services_manager.GetServices( ( HC.TAG_REPOSITORY, ) )
+ file_services = self._controller.services_manager.GetServices( ( HC.FILE_REPOSITORY, ) )
submenu = wx.Menu()
@@ -1388,10 +1394,10 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
repository_admin_permissions = [ ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_CREATE ), ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_OVERRULE ), ( HC.CONTENT_TYPE_ACCOUNT_TYPES, HC.PERMISSION_ACTION_OVERRULE ) ]
- repositories = self._controller.GetServicesManager().GetServices( HC.REPOSITORIES )
+ repositories = self._controller.services_manager.GetServices( HC.REPOSITORIES )
admin_repositories = [ service for service in repositories if True in ( service.HasPermission( content_type, action ) for ( content_type, action ) in repository_admin_permissions ) ]
- servers_admin = self._controller.GetServicesManager().GetServices( ( HC.SERVER_ADMIN, ) )
+ servers_admin = self._controller.services_manager.GetServices( ( HC.SERVER_ADMIN, ) )
server_admins = [ service for service in servers_admin if service.HasPermission( HC.CONTENT_TYPE_SERVICES, HC.PERMISSION_ACTION_OVERRULE ) ]
if len( admin_repositories ) > 0 or len( server_admins ) > 0:
@@ -1582,6 +1588,37 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
with ClientGUIDialogs.DialogGenerateNewAccounts( self, service_key ) as dlg: dlg.ShowModal()
+ def _GetDefaultPageInsertionIndex( self ):
+
+ new_page_goes = self._new_options.GetInteger( 'default_new_page_goes' )
+
+ current_index = self._notebook.GetSelection()
+
+ if current_index == wx.NOT_FOUND:
+
+ new_page_goes = CC.NEW_PAGE_GOES_FAR_LEFT
+
+
+ if new_page_goes == CC.NEW_PAGE_GOES_FAR_LEFT:
+
+ insertion_index = 0
+
+ elif new_page_goes == CC.NEW_PAGE_GOES_LEFT_OF_CURRENT:
+
+ insertion_index = current_index
+
+ elif new_page_goes == CC.NEW_PAGE_GOES_RIGHT_OF_CURRENT:
+
+ insertion_index = current_index + 1
+
+ elif new_page_goes == CC.NEW_PAGE_GOES_FAR_RIGHT:
+
+ insertion_index = self._notebook.GetPageCount()
+
+
+ return insertion_index
+
+
def _ImportFiles( self, paths = None ):
if paths is None: paths = []
@@ -1931,7 +1968,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
return
- service = self._controller.GetServicesManager().GetService( service_key )
+ service = self._controller.services_manager.GetService( service_key )
with ClientGUIDialogs.DialogTextEntry( self, 'Enter the account key for the account to be modified.' ) as dlg:
@@ -1987,7 +2024,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
- def _NewPage( self, page_name, management_controller, initial_media_results = None ):
+ def _NewPage( self, page_name, management_controller, initial_media_results = None, forced_insertion_index = None ):
self._controller.ResetIdleTimer()
self._controller.ResetPageChangeTimer()
@@ -1999,39 +2036,22 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
page = ClientGUIPages.Page( self._notebook, self._controller, management_controller, initial_media_results )
- if self._next_new_page_index is None:
+ if forced_insertion_index is None:
- new_page_goes = self._new_options.GetInteger( 'default_new_page_goes' )
-
- current_index = self._notebook.GetSelection()
-
- if current_index == wx.NOT_FOUND:
+ if self._next_new_page_index is None:
- new_page_goes = CC.NEW_PAGE_GOES_FAR_LEFT
+ insertion_index = self._GetDefaultPageInsertionIndex()
-
- if new_page_goes == CC.NEW_PAGE_GOES_FAR_LEFT:
+ else:
- insertion_index = 0
+ insertion_index = self._next_new_page_index
- elif new_page_goes == CC.NEW_PAGE_GOES_LEFT_OF_CURRENT:
-
- insertion_index = current_index
-
- elif new_page_goes == CC.NEW_PAGE_GOES_RIGHT_OF_CURRENT:
-
- insertion_index = current_index + 1
-
- elif new_page_goes == CC.NEW_PAGE_GOES_FAR_RIGHT:
-
- insertion_index = self._notebook.GetPageCount()
+ self._next_new_page_index = None
else:
- insertion_index = self._next_new_page_index
-
- self._next_new_page_index = None
+ insertion_index = forced_insertion_index
self._notebook.InsertPage( insertion_index, page, page_name, select = True )
@@ -2095,7 +2115,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
management_controller = ClientGUIManagement.CreateManagementControllerPetitions( service_key )
- service = self._controller.GetServicesManager().GetService( service_key )
+ service = self._controller.services_manager.GetService( service_key )
page_name = service.GetName() + ' petitions'
@@ -2113,7 +2133,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
tag_service_key = new_options.GetKey( 'default_tag_service_search_page' )
- if not self._controller.GetServicesManager().ServiceExists( tag_service_key ):
+ if not self._controller.services_manager.ServiceExists( tag_service_key ):
tag_service_key = CC.COMBINED_TAG_SERVICE_KEY
@@ -2265,7 +2285,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
if dlg.ShowModal() == wx.ID_YES:
- self._controller.CallToThread( self._controller.GetClientFilesManager().Rebalance, partial = False )
+ self._controller.CallToThread( self._controller.client_files_manager.Rebalance, partial = False )
@@ -2368,7 +2388,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _RegenerateThumbnails( self ):
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
text = 'This will rebuild all your thumbnails from the original files. You probably only want to do this if you experience thumbnail errors. If you have a lot of files, it will take some time. A popup message will show its progress.'
text += os.linesep * 2
@@ -2556,7 +2576,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
def _StartIPFSDownload( self ):
- ipfs_services = self._controller.GetServicesManager().GetServices( ( HC.IPFS, ) )
+ ipfs_services = self._controller.services_manager.GetServices( ( HC.IPFS, ) )
if len( ipfs_services ) > 0:
@@ -2811,7 +2831,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
def _THREADUploadPending( self, service_key ):
- service = self._controller.GetServicesManager().GetService( service_key )
+ service = self._controller.services_manager.GetService( service_key )
service_name = service.GetName()
service_type = service.GetServiceType()
@@ -2869,7 +2889,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
media_result = result
- client_files_manager = self._controller.GetClientFilesManager()
+ client_files_manager = self._controller.client_files_manager
hash = media_result.GetHash()
mime = media_result.GetMime()
diff --git a/include/ClientGUIACDropdown.py b/include/ClientGUIACDropdown.py
index df563f9b..71496be6 100644
--- a/include/ClientGUIACDropdown.py
+++ b/include/ClientGUIACDropdown.py
@@ -521,9 +521,9 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
self._current_matches = []
- file_service = HG.client_controller.GetServicesManager().GetService( self._file_service_key )
+ file_service = HG.client_controller.services_manager.GetService( self._file_service_key )
- tag_service = HG.client_controller.GetServicesManager().GetService( self._tag_service_key )
+ tag_service = HG.client_controller.services_manager.GetService( self._tag_service_key )
self._file_repo_button = ClientGUICommon.BetterButton( self._dropdown_window, file_service.GetName(), self.FileButtonHit )
self._file_repo_button.SetMinSize( ( 20, -1 ) )
@@ -541,7 +541,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
self._file_service_key = file_service_key
- file_service = HG.client_controller.GetServicesManager().GetService( self._file_service_key )
+ file_service = HG.client_controller.services_manager.GetService( self._file_service_key )
name = file_service.GetName()
@@ -561,7 +561,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
self._dropdown_list.SetTagService( self._tag_service_key )
- tag_service = tag_service = HG.client_controller.GetServicesManager().GetService( self._tag_service_key )
+ tag_service = tag_service = HG.client_controller.services_manager.GetService( self._tag_service_key )
name = tag_service.GetName()
@@ -594,7 +594,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
def FileButtonHit( self ):
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
services = []
@@ -626,7 +626,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
def TagButtonHit( self ):
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
services = []
diff --git a/include/ClientGUICanvas.py b/include/ClientGUICanvas.py
index 5634f5a5..47149584 100755
--- a/include/ClientGUICanvas.py
+++ b/include/ClientGUICanvas.py
@@ -423,7 +423,7 @@ class Animation( wx.Window ):
hash = self._media.GetHash()
mime = self._media.GetMime()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( hash, mime )
@@ -1270,7 +1270,7 @@ class Canvas( wx.Window ):
if self._current_media is not None:
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
paths = [ client_files_manager.GetFilePath( self._current_media.GetHash(), self._current_media.GetMime() ) ]
@@ -1282,7 +1282,7 @@ class Canvas( wx.Window ):
if self._current_media is not None:
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( self._current_media.GetHash(), self._current_media.GetMime() )
@@ -1556,7 +1556,7 @@ class Canvas( wx.Window ):
return
- if len( HG.client_controller.GetServicesManager().GetServices( HC.RATINGS_SERVICES ) ) > 0:
+ if len( HG.client_controller.services_manager.GetServices( HC.RATINGS_SERVICES ) ) > 0:
with ClientGUIDialogsManage.DialogManageRatings( self, ( self._current_media, ) ) as dlg:
@@ -1636,7 +1636,7 @@ class Canvas( wx.Window ):
hash = self._current_media.GetHash()
mime = self._current_media.GetMime()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( hash, mime )
@@ -1736,7 +1736,7 @@ class Canvas( wx.Window ):
try:
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
except HydrusExceptions.DataMissing:
@@ -2439,7 +2439,7 @@ class CanvasPanel( Canvas ):
if self._current_media is not None:
- services = HG.client_controller.GetServicesManager().GetServices()
+ services = HG.client_controller.services_manager.GetServices()
locations_manager = self._current_media.GetLocationsManager()
@@ -2686,7 +2686,7 @@ class CanvasWithDetails( Canvas ):
# ratings
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
like_services = services_manager.GetServices( ( HC.LOCAL_RATING_LIKE, ), randomised = False )
@@ -2847,7 +2847,7 @@ class CanvasWithHovers( CanvasWithDetails ):
self._hover_commands = self._GenerateHoverTopFrame()
self._hover_tags = ClientGUIHoverFrames.FullscreenHoverFrameTags( self, self._canvas_key )
- ratings_services = HG.client_controller.GetServicesManager().GetServices( ( HC.RATINGS_SERVICES ) )
+ ratings_services = HG.client_controller.services_manager.GetServices( ( HC.RATINGS_SERVICES ) )
if len( ratings_services ) > 0:
@@ -4708,7 +4708,7 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ):
if self._current_media is not None:
- services = HG.client_controller.GetServicesManager().GetServices()
+ services = HG.client_controller.services_manager.GetServices()
local_ratings_services = [ service for service in services if service.GetServiceType() in ( HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ) ]
@@ -4983,7 +4983,7 @@ class MediaContainer( wx.Window ):
raise Exception( 'Failed to initialise the flash window' )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
self._media_window.movie = client_files_manager.GetFilePath( self._media.GetHash(), HC.APPLICATION_FLASH )
@@ -5422,7 +5422,7 @@ class EmbedButton( wx.Window ):
hash = self._media.GetHash()
- thumbnail_path = HG.client_controller.GetClientFilesManager().GetFullSizeThumbnailPath( hash )
+ thumbnail_path = HG.client_controller.client_files_manager.GetFullSizeThumbnailPath( hash )
self._thumbnail_bmp = ClientRendering.GenerateHydrusBitmap( thumbnail_path ).GetWxBitmap()
@@ -5450,7 +5450,7 @@ class OpenExternallyPanel( wx.Panel ):
hash = self._media.GetHash()
- thumbnail_path = HG.client_controller.GetClientFilesManager().GetFullSizeThumbnailPath( hash )
+ thumbnail_path = HG.client_controller.client_files_manager.GetFullSizeThumbnailPath( hash )
bmp = ClientRendering.GenerateHydrusBitmap( thumbnail_path ).GetWxBitmap()
@@ -5480,7 +5480,7 @@ class OpenExternallyPanel( wx.Panel ):
hash = self._media.GetHash()
mime = self._media.GetMime()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( hash, mime )
diff --git a/include/ClientGUICommon.py b/include/ClientGUICommon.py
index 53dc0657..4f1ebd22 100755
--- a/include/ClientGUICommon.py
+++ b/include/ClientGUICommon.py
@@ -482,7 +482,7 @@ class CheckboxCollect( wx.combo.ComboCtrl ):
text_and_data_tuples = list( [ ( namespace, ( 'namespace', namespace ) ) for namespace in text_and_data_tuples ] )
text_and_data_tuples.sort()
- ratings_services = HG.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ) )
+ ratings_services = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ) )
for ratings_service in ratings_services:
@@ -598,7 +598,7 @@ class ChoiceSort( BetterChoice ):
self._page_key = page_key
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
sort_choices = ClientData.GetSortChoices( add_namespaces_and_ratings = add_namespaces_and_ratings )
@@ -2539,7 +2539,7 @@ class RatingLikeCanvas( RatingLike ):
self._current_media = None
self._rating_state = None
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
name = service.GetName()
@@ -2648,7 +2648,7 @@ class RatingNumerical( wx.Window ):
self._service_key = service_key
- self._service = HG.client_controller.GetServicesManager().GetService( self._service_key )
+ self._service = HG.client_controller.services_manager.GetService( self._service_key )
self._num_stars = self._service.GetNumStars()
self._allow_zero = self._service.AllowZero()
diff --git a/include/ClientGUIDialogs.py b/include/ClientGUIDialogs.py
index 9836fae9..15f8c977 100755
--- a/include/ClientGUIDialogs.py
+++ b/include/ClientGUIDialogs.py
@@ -61,7 +61,7 @@ def SelectServiceKey( service_types = HC.ALL_SERVICES, service_keys = None, unal
if service_keys is None:
- services = HG.client_controller.GetServicesManager().GetServices( service_types )
+ services = HG.client_controller.services_manager.GetServices( service_types )
service_keys = [ service.GetServiceKey() for service in services ]
@@ -83,7 +83,7 @@ def SelectServiceKey( service_types = HC.ALL_SERVICES, service_keys = None, unal
else:
- services = { HG.client_controller.GetServicesManager().GetService( service_key ) for service_key in service_keys }
+ services = { HG.client_controller.services_manager.GetService( service_key ) for service_key in service_keys }
list_of_tuples = [ ( service.GetName(), service.GetServiceKey() ) for service in services ]
@@ -372,7 +372,7 @@ class DialogGenerateNewAccounts( Dialog ):
self._num.SetValue( 1 )
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
response = service.Request( HC.GET, 'account_types' )
@@ -439,7 +439,7 @@ class DialogGenerateNewAccounts( Dialog ):
expires = HydrusData.GetNow() + lifetime
- service = HG.client_controller.GetServicesManager().GetService( self._service_key )
+ service = HG.client_controller.services_manager.GetService( self._service_key )
try:
@@ -570,7 +570,7 @@ class DialogInputFileSystemPredicates( Dialog ):
elif predicate_type == HC.PREDICATE_TYPE_SYSTEM_RATING:
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
ratings_services = services_manager.GetServices( ( HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ) )
@@ -778,7 +778,7 @@ class DialogInputLocalBooruShare( Dialog ):
def EventCopyExternalShareURL( self, event ):
- self._service = HG.client_controller.GetServicesManager().GetService( CC.LOCAL_BOORU_SERVICE_KEY )
+ self._service = HG.client_controller.services_manager.GetService( CC.LOCAL_BOORU_SERVICE_KEY )
external_ip = HydrusNATPunch.GetExternalIP() # eventually check for optional host replacement here
@@ -796,7 +796,7 @@ class DialogInputLocalBooruShare( Dialog ):
def EventCopyInternalShareURL( self, event ):
- self._service = HG.client_controller.GetServicesManager().GetService( CC.LOCAL_BOORU_SERVICE_KEY )
+ self._service = HG.client_controller.services_manager.GetService( CC.LOCAL_BOORU_SERVICE_KEY )
internal_ip = '127.0.0.1'
@@ -1606,7 +1606,7 @@ class DialogModifyAccounts( Dialog ):
Dialog.__init__( self, parent, 'modify account' )
- self._service = HG.client_controller.GetServicesManager().GetService( service_key )
+ self._service = HG.client_controller.services_manager.GetService( service_key )
self._subject_identifiers = list( subject_identifiers )
#
@@ -1828,7 +1828,7 @@ class DialogPageChooser( Dialog ):
self.SetInitialSize( ( 420, 210 ) )
- self._services = HG.client_controller.GetServicesManager().GetServices()
+ self._services = HG.client_controller.services_manager.GetServices()
repository_petition_permissions = [ ( content_type, HC.PERMISSION_ACTION_OVERRULE ) for content_type in HC.REPOSITORY_CONTENT_TYPES ]
@@ -1862,7 +1862,7 @@ class DialogPageChooser( Dialog ):
elif entry_type in ( 'page_query', 'page_petitions' ):
- name = HG.client_controller.GetServicesManager().GetService( obj ).GetName()
+ name = HG.client_controller.services_manager.GetService( obj ).GetName()
button.SetLabelText( name )
@@ -2119,7 +2119,7 @@ class DialogPathsToTags( Dialog ):
#
- services = HG.client_controller.GetServicesManager().GetServices( ( HC.TAG_REPOSITORY, ) )
+ services = HG.client_controller.services_manager.GetServices( ( HC.TAG_REPOSITORY, ) )
for service in services:
@@ -3393,7 +3393,7 @@ class DialogSetupExport( Dialog ):
self._tags_box = ClientGUICommon.StaticBoxSorterForListBoxTags( self, 'files\' tags' )
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
self._neighbouring_txt_tag_service_keys = services_manager.FilterValidServiceKeys( new_options.GetKeyList( 'default_neighbouring_txt_tag_service_keys' ) )
@@ -3580,7 +3580,7 @@ class DialogSetupExport( Dialog ):
terms = ClientExporting.ParseExportPhrase( pattern )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
self._export.Disable()
@@ -3650,7 +3650,7 @@ class DialogSetupExport( Dialog ):
def EventExportTagTxtsChanged( self, event ):
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
tag_services = services_manager.GetServices( HC.TAG_SERVICES )
diff --git a/include/ClientGUIDialogsManage.py b/include/ClientGUIDialogsManage.py
index 12cb73ec..dd975437 100644
--- a/include/ClientGUIDialogsManage.py
+++ b/include/ClientGUIDialogsManage.py
@@ -2534,7 +2534,7 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
self._txt_parse_st = wx.StaticText( self._tag_box, label = '' )
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
self._txt_parse_tag_service_keys = services_manager.FilterValidServiceKeys( txt_parse_tag_service_keys )
@@ -2730,7 +2730,7 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
def _RefreshTxtParseText( self ):
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
services = [ services_manager.GetService( service_key ) for service_key in self._txt_parse_tag_service_keys ]
@@ -2772,7 +2772,7 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
def EventEditTxtParsing( self, event ):
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
tag_services = services_manager.GetServices( HC.TAG_SERVICES )
@@ -3042,8 +3042,8 @@ class DialogManageRatings( ClientGUIDialogs.Dialog ):
#
- like_services = HG.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_LIKE, ), randomised = False )
- numerical_services = HG.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_NUMERICAL, ), randomised = False )
+ like_services = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_RATING_LIKE, ), randomised = False )
+ numerical_services = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_RATING_NUMERICAL, ), randomised = False )
self._panels = []
@@ -3502,7 +3502,7 @@ class DialogManageTagCensorship( ClientGUIDialogs.Dialog ):
#
- services = HG.client_controller.GetServicesManager().GetServices( ( HC.COMBINED_TAG, HC.TAG_REPOSITORY, HC.LOCAL_TAG ) )
+ services = HG.client_controller.services_manager.GetServices( ( HC.COMBINED_TAG, HC.TAG_REPOSITORY, HC.LOCAL_TAG ) )
for service in services:
@@ -3668,7 +3668,7 @@ class DialogManageTagParents( ClientGUIDialogs.Dialog ):
#
- services = HG.client_controller.GetServicesManager().GetServices( ( HC.TAG_REPOSITORY, ) )
+ services = HG.client_controller.services_manager.GetServices( ( HC.TAG_REPOSITORY, ) )
for service in services:
@@ -3769,15 +3769,9 @@ class DialogManageTagParents( ClientGUIDialogs.Dialog ):
if service_key != CC.LOCAL_TAG_SERVICE_KEY:
- self._service = HG.client_controller.GetServicesManager().GetService( service_key )
+ self._service = HG.client_controller.services_manager.GetService( service_key )
- self._original_statuses_to_pairs = HG.client_controller.Read( 'tag_parents', service_key )
-
- self._current_statuses_to_pairs = collections.defaultdict( set )
-
- self._current_statuses_to_pairs.update( { key : set( value ) for ( key, value ) in self._original_statuses_to_pairs.items() } )
-
self._pairs_to_reasons = {}
self._tag_parents = ClientGUICommon.SaneListCtrl( self, 250, [ ( '', 30 ), ( 'child', 160 ), ( 'parent', -1 ) ] )
@@ -3791,7 +3785,10 @@ class DialogManageTagParents( ClientGUIDialogs.Dialog ):
expand_parents = True
self._child_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterChildren, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key )
+ self._child_input.Disable()
+
self._parent_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterParents, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key )
+ self._parent_input.Disable()
self._add = wx.Button( self, label = 'add' )
self._add.Bind( wx.EVT_BUTTON, self.EventAddButton )
@@ -3799,36 +3796,9 @@ class DialogManageTagParents( ClientGUIDialogs.Dialog ):
#
- petitioned_pairs = set( self._original_statuses_to_pairs[ HC.CONTENT_STATUS_PETITIONED ] )
-
- for ( status, pairs ) in self._original_statuses_to_pairs.items():
-
- if status != HC.CONTENT_STATUS_DELETED:
-
- sign = HydrusData.ConvertStatusToPrefix( status )
-
- for ( child, parent ) in pairs:
-
- if status == HC.CONTENT_STATUS_CURRENT and ( child, parent ) in petitioned_pairs:
-
- continue
-
-
- self._tag_parents.Append( ( sign, child, parent ), ( status, child, parent ) )
-
-
-
-
- self._tag_parents.SortListItems( 2 )
-
- if tags is not None:
-
- self.EnterChildren( tags )
-
-
#
- intro = 'Files with a tag on the left will also be given the tag on the right.'
+ self._status_st = ClientGUICommon.BetterStaticText( self, u'initialising\u2026' )
tags_box = wx.BoxSizer( wx.HORIZONTAL )
@@ -3842,7 +3812,7 @@ class DialogManageTagParents( ClientGUIDialogs.Dialog ):
vbox = wx.BoxSizer( wx.VERTICAL )
- vbox.AddF( ClientGUICommon.BetterStaticText( self, intro ), CC.FLAGS_EXPAND_PERPENDICULAR )
+ vbox.AddF( self._status_st, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._tag_parents, CC.FLAGS_EXPAND_BOTH_WAYS )
vbox.AddF( self._add, CC.FLAGS_LONE_BUTTON )
vbox.AddF( tags_box, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
@@ -3850,6 +3820,10 @@ class DialogManageTagParents( ClientGUIDialogs.Dialog ):
self.SetSizer( vbox )
+ #
+
+ HG.client_controller.CallToThread( self.THREADInitialise, tags )
+
def _AddPairs( self, children, parent ):
@@ -4231,6 +4205,57 @@ class DialogManageTagParents( ClientGUIDialogs.Dialog ):
else: self._parent_input.SetFocus()
+ def THREADInitialise( self, tags ):
+
+ def wx_code():
+
+ if not self:
+
+ return
+
+
+ self._status_st.SetLabelText( 'Files with a tag on the left will also be given the tag on the right.' )
+
+ self._child_input.Enable()
+ self._parent_input.Enable()
+
+ petitioned_pairs = set( self._original_statuses_to_pairs[ HC.CONTENT_STATUS_PETITIONED ] )
+
+ for ( status, pairs ) in self._original_statuses_to_pairs.items():
+
+ if status != HC.CONTENT_STATUS_DELETED:
+
+ sign = HydrusData.ConvertStatusToPrefix( status )
+
+ for ( child, parent ) in pairs:
+
+ if status == HC.CONTENT_STATUS_CURRENT and ( child, parent ) in petitioned_pairs:
+
+ continue
+
+
+ self._tag_parents.Append( ( sign, child, parent ), ( status, child, parent ) )
+
+
+
+
+ self._tag_parents.SortListItems( 2 )
+
+ if tags is not None:
+
+ self.EnterChildren( tags )
+
+
+
+ self._original_statuses_to_pairs = HG.client_controller.Read( 'tag_parents', self._service_key )
+
+ self._current_statuses_to_pairs = collections.defaultdict( set )
+
+ self._current_statuses_to_pairs.update( { key : set( value ) for ( key, value ) in self._original_statuses_to_pairs.items() } )
+
+ wx.CallAfter( wx_code )
+
+
class DialogManageTagSiblings( ClientGUIDialogs.Dialog ):
@@ -4256,7 +4281,7 @@ class DialogManageTagSiblings( ClientGUIDialogs.Dialog ):
self._tag_repositories.AddPage( name, name, page )
- services = HG.client_controller.GetServicesManager().GetServices( ( HC.TAG_REPOSITORY, ) )
+ services = HG.client_controller.services_manager.GetServices( ( HC.TAG_REPOSITORY, ) )
for service in services:
@@ -4351,15 +4376,9 @@ class DialogManageTagSiblings( ClientGUIDialogs.Dialog ):
if self._service_key != CC.LOCAL_TAG_SERVICE_KEY:
- self._service = HG.client_controller.GetServicesManager().GetService( service_key )
+ self._service = HG.client_controller.services_manager.GetService( service_key )
- self._original_statuses_to_pairs = HG.client_controller.Read( 'tag_siblings', service_key )
-
- self._current_statuses_to_pairs = collections.defaultdict( set )
-
- self._current_statuses_to_pairs.update( { key : set( value ) for ( key, value ) in self._original_statuses_to_pairs.items() } )
-
self._pairs_to_reasons = {}
self._current_new = None
@@ -4375,7 +4394,10 @@ class DialogManageTagSiblings( ClientGUIDialogs.Dialog ):
expand_parents = False
self._old_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterOlds, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key )
+ self._old_input.Disable()
+
self._new_input = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.SetNew, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key )
+ self._new_input.Disable()
self._add = wx.Button( self, label = 'add' )
self._add.Bind( wx.EVT_BUTTON, self.EventAddButton )
@@ -4383,36 +4405,7 @@ class DialogManageTagSiblings( ClientGUIDialogs.Dialog ):
#
- petitioned_pairs = set( self._original_statuses_to_pairs[ HC.CONTENT_STATUS_PETITIONED ] )
-
- for ( status, pairs ) in self._original_statuses_to_pairs.items():
-
- if status != HC.CONTENT_STATUS_DELETED:
-
- sign = HydrusData.ConvertStatusToPrefix( status )
-
- for ( old, new ) in pairs:
-
- if status == HC.CONTENT_STATUS_CURRENT and ( old, new ) in petitioned_pairs:
-
- continue
-
-
- self._tag_siblings.Append( ( sign, old, new ), ( status, old, new ) )
-
-
-
-
- self._tag_siblings.SortListItems( 2 )
-
- if tags is not None:
-
- self.EnterOlds( tags )
-
-
- #
-
- intro = 'Tags on the left will be replaced by those on the right.'
+ self._status_st = ClientGUICommon.BetterStaticText( self, u'initialising\u2026' )
new_sibling_box = wx.BoxSizer( wx.VERTICAL )
@@ -4432,7 +4425,7 @@ class DialogManageTagSiblings( ClientGUIDialogs.Dialog ):
vbox = wx.BoxSizer( wx.VERTICAL )
- vbox.AddF( ClientGUICommon.BetterStaticText( self, intro ), CC.FLAGS_EXPAND_PERPENDICULAR )
+ vbox.AddF( self._status_st, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._tag_siblings, CC.FLAGS_EXPAND_BOTH_WAYS )
vbox.AddF( self._add, CC.FLAGS_LONE_BUTTON )
vbox.AddF( text_box, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
@@ -4440,6 +4433,9 @@ class DialogManageTagSiblings( ClientGUIDialogs.Dialog ):
self.SetSizer( vbox )
+ #
+
+ HG.client_controller.CallToThread( self.THREADInitialise, tags )
def _AddPairs( self, olds, new ):
@@ -4893,6 +4889,57 @@ class DialogManageTagSiblings( ClientGUIDialogs.Dialog ):
else: self._new_input.SetFocus()
+ def THREADInitialise( self, tags ):
+
+ def wx_code():
+
+ if not self:
+
+ return
+
+
+ self._status_st.SetLabelText( 'Tags on the left will be replaced by those on the right.' )
+
+ self._old_input.Enable()
+ self._new_input.Enable()
+
+ petitioned_pairs = set( self._original_statuses_to_pairs[ HC.CONTENT_STATUS_PETITIONED ] )
+
+ for ( status, pairs ) in self._original_statuses_to_pairs.items():
+
+ if status != HC.CONTENT_STATUS_DELETED:
+
+ sign = HydrusData.ConvertStatusToPrefix( status )
+
+ for ( old, new ) in pairs:
+
+ if status == HC.CONTENT_STATUS_CURRENT and ( old, new ) in petitioned_pairs:
+
+ continue
+
+
+ self._tag_siblings.Append( ( sign, old, new ), ( status, old, new ) )
+
+
+
+
+ self._tag_siblings.SortListItems( 2 )
+
+ if tags is not None:
+
+ self.EnterOlds( tags )
+
+
+
+ self._original_statuses_to_pairs = HG.client_controller.Read( 'tag_siblings', self._service_key )
+
+ self._current_statuses_to_pairs = collections.defaultdict( set )
+
+ self._current_statuses_to_pairs.update( { key : set( value ) for ( key, value ) in self._original_statuses_to_pairs.items() } )
+
+ wx.CallAfter( wx_code )
+
+
class DialogManageUPnP( ClientGUIDialogs.Dialog ):
diff --git a/include/ClientGUIHoverFrames.py b/include/ClientGUIHoverFrames.py
index a5cf99da..2f012fec 100644
--- a/include/ClientGUIHoverFrames.py
+++ b/include/ClientGUIHoverFrames.py
@@ -724,7 +724,7 @@ class FullscreenHoverFrameTopRight( FullscreenHoverFrame ):
like_hbox.AddF( ( 16, 16 ), CC.FLAGS_EXPAND_BOTH_WAYS )
- like_services = HG.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_LIKE, ), randomised = False )
+ like_services = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_RATING_LIKE, ), randomised = False )
for service in like_services:
@@ -739,7 +739,7 @@ class FullscreenHoverFrameTopRight( FullscreenHoverFrame ):
vbox.AddF( like_hbox, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
- numerical_services = HG.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_NUMERICAL, ), randomised = False )
+ numerical_services = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_RATING_NUMERICAL, ), randomised = False )
for service in numerical_services:
diff --git a/include/ClientGUIManagement.py b/include/ClientGUIManagement.py
index 9521780d..5e82da76 100755
--- a/include/ClientGUIManagement.py
+++ b/include/ClientGUIManagement.py
@@ -136,7 +136,7 @@ def CreateManagementControllerImportURLs():
def CreateManagementControllerPetitions( petition_service_key ):
- petition_service = HG.client_controller.GetServicesManager().GetService( petition_service_key )
+ petition_service = HG.client_controller.services_manager.GetService( petition_service_key )
petition_service_type = petition_service.GetServiceType()
@@ -394,13 +394,13 @@ def GenerateDumpMultipartFormDataCTAndBody( fields ):
def EventRefreshCaptcha( self, event ):
- javascript_string = self._controller.DoHTTP( HC.GET, 'http://www.google.com/recaptcha/api/challenge?k=' + self._captcha_key )
+ javascript_string = self._controller.DoHTTP( HC.GET, 'https://www.google.com/recaptcha/api/challenge?k=' + self._captcha_key )
( trash, rest ) = javascript_string.split( 'challenge : \'', 1 )
( self._captcha_challenge, trash ) = rest.split( '\'', 1 )
- jpeg = self._controller.DoHTTP( HC.GET, 'http://www.google.com/recaptcha/api/image?c=' + self._captcha_challenge )
+ jpeg = self._controller.DoHTTP( HC.GET, 'https://www.google.com/recaptcha/api/image?c=' + self._captcha_challenge )
( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
@@ -562,7 +562,7 @@ class ManagementController( HydrusSerialisable.SerialisableBase ):
if 'file_service' in self._keys:
- if not HG.client_controller.GetServicesManager().ServiceExists( self._keys[ 'file_service' ] ):
+ if not HG.client_controller.services_manager.ServiceExists( self._keys[ 'file_service' ] ):
self._keys[ 'file_service' ] = CC.COMBINED_LOCAL_FILE_SERVICE_KEY
@@ -864,7 +864,7 @@ class ManagementPanelDuplicateFilter( ManagementPanel ):
def _FileDomainButtonHit( self ):
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
services = []
@@ -945,7 +945,7 @@ class ManagementPanelDuplicateFilter( ManagementPanel ):
self._management_controller.SetKey( 'duplicate_filter_file_domain', service_key )
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
service = services_manager.GetService( service_key )
@@ -2879,7 +2879,7 @@ class ManagementPanelPetitions( ManagementPanel ):
ManagementPanel.__init__( self, parent, page, controller, management_controller )
- self._service = self._controller.GetServicesManager().GetService( self._petition_service_key )
+ self._service = self._controller.services_manager.GetService( self._petition_service_key )
self._can_ban = self._service.HasPermission( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_OVERRULE )
service_type = self._service.GetServiceType()
diff --git a/include/ClientGUIMedia.py b/include/ClientGUIMedia.py
index 8860ee50..08fc6a91 100755
--- a/include/ClientGUIMedia.py
+++ b/include/ClientGUIMedia.py
@@ -40,7 +40,7 @@ ID_TIMER_ANIMATION = wx.NewId()
def AddServiceKeyLabelsToMenu( menu, service_keys, phrase ):
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
if len( service_keys ) == 1:
@@ -68,7 +68,7 @@ def AddServiceKeyLabelsToMenu( menu, service_keys, phrase ):
def AddServiceKeysToMenu( event_handler, menu, service_keys, phrase, description, callable ):
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
if len( service_keys ) == 1:
@@ -164,7 +164,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
def _CopyFilesToClipboard( self ):
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
hashes = self._GetSelectedHashes( discriminant = CC.DISCRIMINANT_LOCAL, ordered = True )
@@ -233,7 +233,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
display_media = self._focussed_media.GetDisplayMedia()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( display_media.GetHash(), display_media.GetMime() )
@@ -244,7 +244,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
media_results = self.GenerateMediaResults( discriminant = CC.DISCRIMINANT_LOCAL, selected_media = set( self._selected_media ) )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
paths = []
@@ -266,7 +266,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
( filename, ) = HG.client_controller.Read( 'service_filenames', service_key, { hash } )
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
if service.GetServiceType() == HC.IPFS:
@@ -282,7 +282,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
prefix = ''
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
if service.GetServiceType() == HC.IPFS:
@@ -483,7 +483,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
hash = display_media.GetHash()
mime = display_media.GetMime()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( hash, mime )
@@ -843,7 +843,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
if len( self._selected_media ) > 0:
- if len( HG.client_controller.GetServicesManager().GetServices( HC.RATINGS_SERVICES ) ) > 0:
+ if len( HG.client_controller.services_manager.GetServices( HC.RATINGS_SERVICES ) ) > 0:
flat_media = self._GetSelectedFlatMedia()
@@ -929,7 +929,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
hash = self._focussed_media.GetHash()
mime = self._focussed_media.GetMime()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( hash, mime )
@@ -949,7 +949,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
hash = self._focussed_media.GetHash()
mime = self._focussed_media.GetMime()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( hash, mime )
@@ -966,7 +966,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
if hashes is not None and len( hashes ) > 0:
- remote_service = HG.client_controller.GetServicesManager().GetService( remote_service_key )
+ remote_service = HG.client_controller.services_manager.GetService( remote_service_key )
service_type = remote_service.GetServiceType()
@@ -1454,7 +1454,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, wx.ScrolledWindow ):
if hashes is not None and len( hashes ) > 0:
- ipfs_service = HG.client_controller.GetServicesManager().GetService( file_service_key )
+ ipfs_service = HG.client_controller.services_manager.GetService( file_service_key )
with ClientGUIDialogs.DialogTextEntry( self, 'Enter a note to describe this directory.' ) as dlg:
@@ -1850,7 +1850,7 @@ class MediaPanelThumbnails( MediaPanel ):
if len( self._selected_media ) > 0:
- services = HG.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_TAG, HC.TAG_REPOSITORY, HC.COMBINED_TAG ) )
+ services = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_TAG, HC.TAG_REPOSITORY, HC.COMBINED_TAG ) )
service_keys = [ service.GetServiceKey() for service in services ]
@@ -2345,7 +2345,7 @@ class MediaPanelThumbnails( MediaPanel ):
file_data_object = wx.FileDataObject()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
for hash in hashes:
@@ -2583,7 +2583,7 @@ class MediaPanelThumbnails( MediaPanel ):
advanced_mode = new_options.GetBoolean( 'advanced_mode' )
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
thumbnail = self._GetThumbnailUnderMouse( event )
@@ -2668,7 +2668,7 @@ class MediaPanelThumbnails( MediaPanel ):
multiple_selected = num_selected > 1
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
services = services_manager.GetServices()
@@ -3927,7 +3927,7 @@ class Thumbnail( Selectable ):
# repo icons
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
repo_icon_x = 0
diff --git a/include/ClientGUIOptionsPanels.py b/include/ClientGUIOptionsPanels.py
index 047bad51..2f919066 100644
--- a/include/ClientGUIOptionsPanels.py
+++ b/include/ClientGUIOptionsPanels.py
@@ -308,7 +308,7 @@ class OptionsPanelTags( OptionsPanel ):
self._services_vbox.Clear( True )
- services = HG.client_controller.GetServicesManager().GetServices( HC.TAG_SERVICES, randomised = False )
+ services = HG.client_controller.services_manager.GetServices( HC.TAG_SERVICES, randomised = False )
button_id = 1
diff --git a/include/ClientGUIPanels.py b/include/ClientGUIPanels.py
index 0f110954..a252adbb 100644
--- a/include/ClientGUIPanels.py
+++ b/include/ClientGUIPanels.py
@@ -774,7 +774,7 @@ class ReviewServicePanel( wx.Panel ):
job_key.SetVariable( 'popup_title', 'exporting updates for ' + self._service.GetName() )
HG.client_controller.pub( 'message', job_key )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
for ( i, update_hash ) in enumerate( update_hashes ):
diff --git a/include/ClientGUIPredicates.py b/include/ClientGUIPredicates.py
index f0357efb..17661622 100644
--- a/include/ClientGUIPredicates.py
+++ b/include/ClientGUIPredicates.py
@@ -184,7 +184,7 @@ class PanelPredicateSystemFileService( PanelPredicateSystem ):
self._current_pending = ClientGUICommon.BetterRadioBox( self, choices = [ ( 'currently in', HC.CONTENT_STATUS_CURRENT ), ( 'pending to', HC.CONTENT_STATUS_PENDING ) ], style = wx.RA_SPECIFY_ROWS )
- services = HG.client_controller.GetServicesManager().GetServices( HC.FILE_SERVICES )
+ services = HG.client_controller.services_manager.GetServices( HC.FILE_SERVICES )
choices = [ ( service.GetName(), service.GetServiceKey() ) for service in services ]
@@ -491,7 +491,7 @@ class PanelPredicateSystemRating( PanelPredicateSystem ):
#
- local_like_services = HG.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_LIKE, ), randomised = False )
+ local_like_services = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_RATING_LIKE, ), randomised = False )
self._like_checkboxes_to_info = {}
@@ -523,7 +523,7 @@ class PanelPredicateSystemRating( PanelPredicateSystem ):
#
- local_numerical_services = HG.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_RATING_NUMERICAL, ), randomised = False )
+ local_numerical_services = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_RATING_NUMERICAL, ), randomised = False )
self._numerical_checkboxes_to_info = {}
diff --git a/include/ClientGUIScrolledPanelsEdit.py b/include/ClientGUIScrolledPanelsEdit.py
index 4a613f10..c56d8cc9 100644
--- a/include/ClientGUIScrolledPanelsEdit.py
+++ b/include/ClientGUIScrolledPanelsEdit.py
@@ -179,7 +179,7 @@ class EditDuplicateActionOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
( tag_service_options, rating_service_options, delete_second_file, sync_archive, delete_both_files ) = duplicate_action_options.ToTuple()
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
for ( service_key, action, tag_censor ) in tag_service_options:
@@ -265,7 +265,7 @@ class EditDuplicateActionOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
existing_service_keys.add( service_key )
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
choice_tuples = []
@@ -344,7 +344,7 @@ class EditDuplicateActionOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
existing_service_keys.add( service_key )
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
choice_tuples = []
@@ -548,7 +548,7 @@ class EditDuplicateActionOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
( service_key, action ) = sort_tuple
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
service = services_manager.GetService( service_key )
@@ -563,7 +563,7 @@ class EditDuplicateActionOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
( service_key, action, tag_censor ) = sort_tuple
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
service = services_manager.GetService( service_key )
diff --git a/include/ClientGUIScrolledPanelsManagement.py b/include/ClientGUIScrolledPanelsManagement.py
index dea4e8dd..6bf4ef44 100644
--- a/include/ClientGUIScrolledPanelsManagement.py
+++ b/include/ClientGUIScrolledPanelsManagement.py
@@ -41,7 +41,7 @@ class ManageAccountTypesPanel( ClientGUIScrolledPanels.ManagePanel ):
def __init__( self, parent, service_key ):
- self._admin_service = HG.client_controller.GetServicesManager().GetService( service_key )
+ self._admin_service = HG.client_controller.services_manager.GetService( service_key )
ClientGUIScrolledPanels.ManagePanel.__init__( self, parent )
@@ -260,7 +260,7 @@ class ManageClientServicesPanel( ClientGUIScrolledPanels.ManagePanel ):
#
- all_services = HG.client_controller.GetServicesManager().GetServices()
+ all_services = HG.client_controller.services_manager.GetServices()
for service in all_services:
@@ -2440,7 +2440,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
rows.append( ( 'Main gui title: ', self._main_gui_title ) )
rows.append( ( 'Default session on startup: ', self._default_gui_session ) )
- rows.append( ( 'By default, new pages: ', self._default_new_page_goes ) )
+ rows.append( ( 'By default, new page tabs: ', self._default_new_page_goes ) )
rows.append( ( 'Confirm client exit: ', self._confirm_client_exit ) )
rows.append( ( 'Confirm sending files to trash: ', self._confirm_trash ) )
rows.append( ( 'Confirm sending more than one file to archive or inbox: ', self._confirm_archive ) )
@@ -2586,7 +2586,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._media_zooms.SetValue( ','.join( ( str( media_zoom ) for media_zoom in media_zooms ) ) )
- mimes_in_correct_order = ( HC.IMAGE_JPEG, HC.IMAGE_PNG, HC.IMAGE_GIF, HC.APPLICATION_FLASH, HC.APPLICATION_PDF, HC.APPLICATION_HYDRUS_UPDATE_CONTENT, HC.APPLICATION_HYDRUS_UPDATE_DEFINITIONS, HC.VIDEO_AVI, HC.VIDEO_FLV, HC.VIDEO_MOV, HC.VIDEO_MP4, HC.VIDEO_MKV, HC.VIDEO_MPEG, HC.VIDEO_WEBM, HC.VIDEO_WMV, HC.AUDIO_MP3, HC.AUDIO_OGG, HC.AUDIO_FLAC, HC.AUDIO_WMA )
+ mimes_in_correct_order = ( HC.IMAGE_JPEG, HC.IMAGE_PNG, HC.IMAGE_APNG, HC.IMAGE_GIF, HC.APPLICATION_FLASH, HC.APPLICATION_PDF, HC.APPLICATION_HYDRUS_UPDATE_CONTENT, HC.APPLICATION_HYDRUS_UPDATE_DEFINITIONS, HC.VIDEO_AVI, HC.VIDEO_FLV, HC.VIDEO_MOV, HC.VIDEO_MP4, HC.VIDEO_MKV, HC.VIDEO_MPEG, HC.VIDEO_WEBM, HC.VIDEO_WMV, HC.AUDIO_MP3, HC.AUDIO_OGG, HC.AUDIO_FLAC, HC.AUDIO_WMA )
for mime in mimes_in_correct_order:
@@ -3219,7 +3219,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._suggested_favourites_services.Append( CC.LOCAL_TAG_SERVICE_KEY, CC.LOCAL_TAG_SERVICE_KEY )
- tag_services = HG.client_controller.GetServicesManager().GetServices( ( HC.TAG_REPOSITORY, ) )
+ tag_services = HG.client_controller.services_manager.GetServices( ( HC.TAG_REPOSITORY, ) )
for tag_service in tag_services:
@@ -3280,7 +3280,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._default_tag_service_search_page.Append( 'all known tags', CC.COMBINED_TAG_SERVICE_KEY )
- services = HG.client_controller.GetServicesManager().GetServices( HC.TAG_SERVICES )
+ services = HG.client_controller.services_manager.GetServices( HC.TAG_SERVICES )
for service in services:
@@ -3528,7 +3528,7 @@ class ManageServerServicesPanel( ClientGUIScrolledPanels.ManagePanel ):
def __init__( self, parent, service_key ):
- self._clientside_admin_service = HG.client_controller.GetServicesManager().GetService( service_key )
+ self._clientside_admin_service = HG.client_controller.services_manager.GetService( service_key )
ClientGUIScrolledPanels.ManagePanel.__init__( self, parent )
@@ -4211,7 +4211,7 @@ class ManageShortcutsPanel( ClientGUIScrolledPanels.ManagePanel ):
#
- services = HG.client_controller.GetServicesManager().GetServices( ( HC.LOCAL_TAG, HC.TAG_REPOSITORY, HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ) )
+ services = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_TAG, HC.TAG_REPOSITORY, HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ) )
for service in services:
@@ -4245,7 +4245,7 @@ class ManageShortcutsPanel( ClientGUIScrolledPanels.ManagePanel ):
( service_key, content_type, action, value ) = data
- self._service = HG.client_controller.GetServicesManager().GetService( service_key )
+ self._service = HG.client_controller.services_manager.GetService( service_key )
service_name = self._service.GetName()
service_type = self._service.GetServiceType()
@@ -4528,7 +4528,7 @@ class ManageShortcutsPanel( ClientGUIScrolledPanels.ManagePanel ):
service_key = self._ratings_like_service_keys.GetClientData( selection )
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
self._current_ratings_like_service = service
@@ -4542,7 +4542,7 @@ class ManageShortcutsPanel( ClientGUIScrolledPanels.ManagePanel ):
service_key = self._ratings_numerical_service_keys.GetClientData( selection )
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
self._current_ratings_numerical_service = service
@@ -5104,7 +5104,7 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
#
- services = HG.client_controller.GetServicesManager().GetServices( HC.TAG_SERVICES )
+ services = HG.client_controller.services_manager.GetServices( HC.TAG_SERVICES )
for service in services:
@@ -5313,7 +5313,7 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
if not self._i_am_local_tag_service:
- self._service = HG.client_controller.GetServicesManager().GetService( tag_service_key )
+ self._service = HG.client_controller.services_manager.GetService( tag_service_key )
self._tags_box_sorter = ClientGUICommon.StaticBoxSorterForListBoxTags( self, 'tags' )
diff --git a/include/ClientGUIScrolledPanelsReview.py b/include/ClientGUIScrolledPanelsReview.py
index fc0856cd..0080ec21 100644
--- a/include/ClientGUIScrolledPanelsReview.py
+++ b/include/ClientGUIScrolledPanelsReview.py
@@ -35,7 +35,7 @@ class AdvancedContentUpdatePanel( ClientGUIScrolledPanels.ReviewPanel ):
self._service_key = service_key
self._hashes = hashes
- service = HG.client_controller.GetServicesManager().GetService( self._service_key )
+ service = HG.client_controller.services_manager.GetService( self._service_key )
self._service_name = service.GetName()
@@ -58,7 +58,7 @@ class AdvancedContentUpdatePanel( ClientGUIScrolledPanels.ReviewPanel ):
#
- services = [ service for service in HG.client_controller.GetServicesManager().GetServices( HC.TAG_SERVICES ) if service.GetServiceKey() != self._service_key ]
+ services = [ service for service in HG.client_controller.services_manager.GetServices( HC.TAG_SERVICES ) if service.GetServiceKey() != self._service_key ]
if len( services ) > 0:
@@ -323,7 +323,7 @@ class ReviewServicesPanel( ClientGUIScrolledPanels.ReviewPanel ):
listbook_dict = {}
- services = self._controller.GetServicesManager().GetServices( randomised = False )
+ services = self._controller.services_manager.GetServices( randomised = False )
lb_to_select = None
service_type_name_to_select = None
@@ -436,3 +436,32 @@ class ReviewServicesPanel( ClientGUIScrolledPanels.ReviewPanel ):
self._InitialiseServices()
+class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
+
+ def __init__( self, parent, controller ):
+
+ self._controller = controller
+
+ ClientGUIScrolledPanels.ReviewPanel.__init__( self, parent )
+
+ # a help button, well labelled, that points to the help page
+
+ # current install dir
+ # current db dir
+
+ # current file dir stuff, listctrl
+ # location | portable | current percents: ftr | ideal percents: ftr
+
+ # move the db and all portable client_files locations (provides warning about the shortcut and lets you copy the new location)
+ # this will require a shutdown
+
+ # rebalance files, listctrl
+ # location | portable yes/no | weight | ideal percent
+ # every change here, if valid, is saved immediately
+
+ # store all resized thumbs in sep location
+ # store all full_size thumbs in sep location
+
+ # do rebalance now button, only enabled if there is work to do
+ # should report to a stoppable job_key panel or something. text, gauge, stop button
+
diff --git a/include/ClientImporting.py b/include/ClientImporting.py
index df68b70f..16ca89f2 100644
--- a/include/ClientImporting.py
+++ b/include/ClientImporting.py
@@ -213,7 +213,7 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
gallery.GetFile( temp_path, url, report_hooks = [ self._file_download_hook ] )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
( status, hash ) = client_files_manager.ImportFile( temp_path, import_file_options = self._import_file_options )
@@ -723,7 +723,7 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
raise Exception( 'File failed to copy--see log for error.' )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
( status, hash ) = client_files_manager.ImportFile( temp_path, import_file_options = self._import_file_options )
@@ -1168,7 +1168,7 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
raise Exception( 'File failed to copy--see log for error.' )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
( status, hash ) = client_files_manager.ImportFile( temp_path, import_file_options = self._import_file_options )
@@ -1434,7 +1434,7 @@ class PageOfImagesImport( HydrusSerialisable.SerialisableBase ):
HG.client_controller.DoHTTP( HC.GET, file_url, report_hooks = report_hooks, temp_path = temp_path )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
( status, hash ) = client_files_manager.ImportFile( temp_path, import_file_options = self._import_file_options )
@@ -2364,7 +2364,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
job_key.SetVariable( 'popup_text_1', x_out_of_y + 'importing file' )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
( status, hash ) = client_files_manager.ImportFile( temp_path, import_file_options = self._import_file_options )
@@ -2858,7 +2858,7 @@ class ThreadWatcherImport( HydrusSerialisable.SerialisableBase ):
HG.client_controller.DoHTTP( HC.GET, file_url, report_hooks = report_hooks, temp_path = temp_path )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
( status, hash ) = client_files_manager.ImportFile( temp_path, import_file_options = self._import_file_options )
@@ -3299,7 +3299,7 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
HG.client_controller.DoHTTP( HC.GET, file_url, report_hooks = report_hooks, temp_path = temp_path )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
( status, hash ) = client_files_manager.ImportFile( temp_path, import_file_options = self._import_file_options )
diff --git a/include/ClientLocalServerResources.py b/include/ClientLocalServerResources.py
index 0af0b08e..36109f00 100644
--- a/include/ClientLocalServerResources.py
+++ b/include/ClientLocalServerResources.py
@@ -43,7 +43,7 @@ class HydrusResourceBooruFile( HydrusResourceBooru ):
local_booru_manager.CheckFileAuthorised( share_key, hash )
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( hash )
@@ -244,7 +244,7 @@ class HydrusResourceBooruThumbnail( HydrusResourceBooru ):
if mime in HC.MIMES_WITH_THUMBNAILS:
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFullSizeThumbnailPath( hash )
@@ -265,7 +265,7 @@ class HydrusResourceLocalFile( HydrusServerResources.HydrusResource ):
hash = request.hydrus_args[ 'hash' ]
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( hash )
@@ -280,7 +280,7 @@ class HydrusResourceLocalThumbnail( HydrusServerResources.HydrusResource ):
hash = request.hydrus_args[ 'hash' ]
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFullSizeThumbnailPath( hash )
diff --git a/include/ClientMedia.py b/include/ClientMedia.py
index 8e2e41e7..1e21b36a 100644
--- a/include/ClientMedia.py
+++ b/include/ClientMedia.py
@@ -54,7 +54,7 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
score += 2
- elif size_ratio > 1.0:
+ elif size_ratio > 1.05:
statements.append( 'This has a larger filesize.' )
@@ -66,7 +66,7 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
score -= 2
- elif size_ratio < 1.0:
+ elif size_ratio < 0.95:
statements.append( 'This has a smaller filesize.' )
@@ -396,7 +396,7 @@ class LocationsManager( object ):
pending = self.GetPendingRemote()
petitioned = self.GetPetitionedRemote()
- remote_services = HG.client_controller.GetServicesManager().GetServices( ( HC.FILE_REPOSITORY, HC.IPFS ) )
+ remote_services = HG.client_controller.services_manager.GetServices( ( HC.FILE_REPOSITORY, HC.IPFS ) )
remote_services = list( remote_services )
@@ -627,7 +627,7 @@ class MediaList( object ):
namespaces_to_collect_by = [ data for ( collect_by_type, data ) in collect_by if collect_by_type == 'namespace' ]
ratings_to_collect_by = [ data for ( collect_by_type, data ) in collect_by if collect_by_type == 'rating' ]
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
for media in medias:
@@ -739,7 +739,7 @@ class MediaList( object ):
elif sort_by_data in ( CC.SORT_BY_OLDEST, CC.SORT_BY_NEWEST ):
- file_service = HG.client_controller.GetServicesManager().GetService( self._file_service_key )
+ file_service = HG.client_controller.services_manager.GetService( self._file_service_key )
file_service_type = file_service.GetServiceType()
@@ -1228,7 +1228,7 @@ class MediaList( object ):
if action == HC.CONTENT_UPDATE_DELETE:
- local_file_domains = HG.client_controller.GetServicesManager().GetServiceKeys( ( HC.LOCAL_FILE_DOMAIN, ) )
+ local_file_domains = HG.client_controller.services_manager.GetServiceKeys( ( HC.LOCAL_FILE_DOMAIN, ) )
non_trash_local_file_services = list( local_file_domains ) + [ CC.COMBINED_LOCAL_FILE_SERVICE_KEY ]
@@ -1729,7 +1729,7 @@ class MediaSingleton( Media ):
timestamp = locations_manager.GetTimestamp( service_key )
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
service_type = service.GetServiceType()
@@ -1895,7 +1895,7 @@ class MediaResult( object ):
def DeletePending( self, service_key ):
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
service_type = service.GetServiceType()
@@ -1983,7 +1983,7 @@ class MediaResult( object ):
( data_type, action, row ) = content_update.ToTuple()
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
service_type = service.GetServiceType()
diff --git a/include/ClientNetworking.py b/include/ClientNetworking.py
index fa7d920d..249ab789 100644
--- a/include/ClientNetworking.py
+++ b/include/ClientNetworking.py
@@ -1,4 +1,6 @@
+import ClientConstants as CC
import collections
+import cPickle
import cStringIO
import HydrusConstants as HC
import HydrusExceptions
@@ -82,14 +84,10 @@ def CheckHydrusVersion( service_key, service_type, response_headers ):
raise HydrusExceptions.NetworkVersionException( 'Network version mismatch! The server\'s network version was ' + str( network_version ) + ', whereas your client\'s is ' + str( HC.NETWORK_VERSION ) + '! ' + message )
-def ConvertURLIntoDomains( url ):
+def ConvertDomainIntoAllApplicableDomains( domain ):
domains = []
- parser_result = urlparse.urlparse( url )
-
- domain = parser_result.netloc
-
while domain.count( '.' ) > 0:
domains.append( domain )
@@ -99,6 +97,14 @@ def ConvertURLIntoDomains( url ):
return domains
+def ConvertURLIntoDomain( url ):
+
+ parser_result = urlparse.urlparse( url )
+
+ domain = HydrusData.ToByteString( parser_result.netloc )
+
+ return domain
+
def RequestsGet( url, params = None, stream = False, headers = None ):
if headers is None:
@@ -1022,7 +1028,7 @@ class HTTPConnection( object ):
return self._DealWithResponse( method, response, parsed_response, size_of_response )
-class BandwidthManager( HydrusSerialisable.SerialisableBase ):
+class NetworkBandwidthManager( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_BANDWIDTH_MANAGER
SERIALISABLE_VERSION = 1
@@ -1031,39 +1037,27 @@ class BandwidthManager( HydrusSerialisable.SerialisableBase ):
HydrusSerialisable.SerialisableBase.__init__( self )
+ self.engine = None
+
self._lock = threading.Lock()
- self._global_bandwidth_tracker = HydrusNetworking.BandwidthTracker()
- self._global_bandwidth_rules = HydrusNetworking.BandwidthRules()
+ self._network_contexts_to_bandwidth_trackers = collections.defaultdict( HydrusNetworking.BandwidthTracker )
+ self._network_contexts_to_bandwidth_rules = {}
- self._domains_to_bandwidth_trackers = collections.defaultdict( HydrusNetworking.BandwidthTracker )
- self._domains_to_bandwidth_rules = {}
+ for context_type in [ CC.NETWORK_CONTEXT_GLOBAL, CC.NETWORK_CONTEXT_HYDRUS, CC.NETWORK_CONTEXT_DOMAIN, CC.NETWORK_CONTEXT_DOWNLOADER, CC.NETWORK_CONTEXT_SUBSCRIPTION ]:
+
+ self._network_contexts_to_bandwidth_rules[ NetworkContext( context_type ) ] = HydrusNetworking.BandwidthRules()
+
- def _GetApplicableTrackersAndRules( self, url = None ):
+ def _GetRules( self, network_context ):
- result = []
-
- if url is not None:
+ if network_context not in self._network_contexts_to_bandwidth_rules:
- domains = ConvertURLIntoDomains( url )
-
- for domain in domains:
-
- if domain in self._domains_to_bandwidth_rules:
-
- bandwidth_tracker = self._domains_to_bandwidth_trackers[ domain ]
-
- bandwidth_rules = self._domains_to_bandwidth_rules[ domain ]
-
- result.append( ( bandwidth_tracker, bandwidth_rules ) )
-
-
+ network_context = NetworkContext( network_context.context_type ) # i.e. the default
- result.append( ( self._global_bandwidth_tracker, self._global_bandwidth_rules ) )
-
- return result
+ return self._network_contexts_to_bandwidth_rules[ network_context ]
def _GetSerialisableInfo( self ):
@@ -1071,8 +1065,8 @@ class BandwidthManager( HydrusSerialisable.SerialisableBase ):
serialisable_global_tracker = self._global_bandwidth_tracker.GetSerialisableTuple()
serialisable_global_rules = self._global_bandwidth_rules.GetSerialisableTuple()
- all_serialisable_trackers = [ ( domain, tracker.GetSerialisableTuple() ) for ( domain, tracker ) in self._domains_to_bandwidth_trackers ]
- all_serialisable_rules = [ ( domain, rules.GetSerialisableTuple() ) for ( domain, rules ) in self._domains_to_bandwidth_rules ]
+ all_serialisable_trackers = [ ( domain, tracker.GetSerialisableTuple() ) for ( domain, tracker ) in self._network_contexts_to_bandwidth_trackers ]
+ all_serialisable_rules = [ ( domain, rules.GetSerialisableTuple() ) for ( domain, rules ) in self._network_contexts_to_bandwidth_rules ]
return ( serialisable_global_tracker, serialisable_global_rules, all_serialisable_trackers, all_serialisable_rules )
@@ -1086,25 +1080,44 @@ class BandwidthManager( HydrusSerialisable.SerialisableBase ):
for ( domain, serialisable_tracker ) in all_serialisable_trackers:
- self._domains_to_bandwidth_trackers[ domain ] = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_tracker )
+ self._network_contexts_to_bandwidth_trackers[ domain ] = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_tracker )
for ( domain, serialisable_rules ) in all_serialisable_rules:
- self._domains_to_bandwidth_rules[ domain ] = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_rules )
+ self._network_contexts_to_bandwidth_rules[ domain ] = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_rules )
- def CanStartGlobally( self ):
-
- return self.CanStartURL( None )
-
-
- def CanStartURL( self, url ):
+ def CanContinue( self, network_contexts ):
with self._lock:
- for ( bandwidth_tracker, bandwidth_rules ) in self._GetApplicableTrackersAndRules( url ):
+ for network_context in network_contexts:
+
+ bandwidth_rules = self._GetRules( network_context )
+
+ bandwidth_tracker = self._network_contexts_to_bandwidth_trackers[ network_context ]
+
+ if not bandwidth_rules.CanContinue( bandwidth_tracker ):
+
+ return False
+
+
+
+ return True
+
+
+
+ def CanStart( self, network_contexts ):
+
+ with self._lock:
+
+ for network_context in network_contexts:
+
+ bandwidth_rules = self._GetRules( network_context )
+
+ bandwidth_tracker = self._network_contexts_to_bandwidth_trackers[ network_context ]
if not bandwidth_rules.CanStart( bandwidth_tracker ):
@@ -1115,88 +1128,174 @@ class BandwidthManager( HydrusSerialisable.SerialisableBase ):
return True
+
+ def DeleteRules( self, network_context ):
+
+ with self._lock:
+
+ if network_context.content_data is None:
+
+ return # can't delete 'default' network contexts
+
+ else:
+
+ if network_context in self._network_contexts_to_bandwidth_rules:
+
+ del self._network_contexts_to_bandwidth_rules[ network_context ]
+
+
+
- def GetEstimateInfo( self, domain = None ):
+ def GetDomains( self, history_time_delta_threshold = 86400 * 30 ):
+
+ with self._lock:
+
+ domains = []
+
+ for ( network_context, bandwidth_tracker ) in self._network_contexts_to_bandwidth_trackers.items():
+
+ if network_context.context_type == CC.NETWORK_CONTEXT_DOMAIN and bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, history_time_delta_threshold ) > 0:
+
+ domains.append( network_context.content_data )
+
+
+
+ return domains
+
+
+
+ def GetEstimateInfo( self, network_contexts ):
with self._lock:
# something that returns ( 'about a minute until you can request again', 60 )
+ # figure out the longest estimate from the rules and trackers
+
+ # make some pretty text out of that
+
+ # return ( text, seconds )
+
pass
- def GetDomainsAndTrackers( self ):
+ def GetTracker( self, network_context ):
with self._lock:
- result = list( self._domains_to_bandwidth_trackers.items() )
-
- result.sort()
-
- result.insert( 0, ( 'global', self._global_bandwidth_tracker ) )
-
- return result
+ return self._network_contexts_to_bandwidth_trackers[ network_context ]
- def ReportDataUsedGlobally( self, num_bytes ):
-
- self.ReportDataUsedURL( None, num_bytes )
-
-
- def ReportDataUsedURL( self, url, num_bytes ):
+ def ReportDataUsed( self, network_contexts, num_bytes ):
with self._lock:
- for ( bandwidth_tracker, bandwidth_rules ) in self._GetApplicableTrackersAndRules( url ):
+ for network_context in network_contexts:
- bandwidth_tracker.ReportDataUsed( num_bytes )
+ self._network_contexts_to_bandwidth_trackers[ network_context ].ReportDataUsed( num_bytes )
- def ReportRequestUsedGlobally( self ):
-
- self.ReportRequestUsedURL( None )
-
-
- def ReportRequestUsedURL( self, url ):
+ def ReportRequestUsed( self, network_contexts ):
with self._lock:
- for ( bandwidth_tracker, bandwidth_rules ) in self._GetApplicableTrackersAndRules( url ):
+ for network_context in network_contexts:
- bandwidth_tracker.ReportRequestUsed()
+ self._network_contexts_to_bandwidth_trackers[ network_context ].ReportRequestUsed()
- def SetRules( self, domain, bandwidth_rules ):
+ def SetRules( self, network_context, bandwidth_rules ):
with self._lock:
- if domain is None:
-
- self._global_bandwidth_rules = bandwidth_rules
-
- else:
-
- self._domains_to_bandwidth_rules[ domain ] = bandwidth_rules
-
+ self._network_contexts_to_bandwidth_rules[ network_context ] = bandwidth_rules
-HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_BANDWIDTH_MANAGER ] = BandwidthManager
+HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_BANDWIDTH_MANAGER ] = NetworkBandwidthManager
+
+class NetworkContext( HydrusSerialisable.SerialisableBase ):
+
+ SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_CONTEXT
+ SERIALISABLE_VERSION = 1
+
+ def __init__( self, context_type = None, context_data = None ):
+
+ HydrusSerialisable.SerialisableBase.__init__( self )
+
+ self.context_type = context_type
+ self.context_data = context_data
+
+
+ def __eq__( self, other ):
+
+ return self.__hash__() == other.__hash__()
+
+
+ def __hash__( self ):
+
+ return ( self.context_type, self.context_data ).__hash__()
+
+
+ def __ne__( self, other ):
+
+ return self.__hash__() != other.__hash__()
+
+
+ def _GetSerialisableInfo( self ):
+
+ if self.context_data is None:
+
+ serialisable_context_data = self.context_data.encode( 'hex' )
+
+ else:
+
+ serialisable_context_data = self.context_data
+
+
+ return ( self.context_type, self.context_data )
+
+
+ def _InitialiseFromSerialisableInfo( self, serialisable_info ):
+
+ ( self.context_type, serialisable_context_data ) = serialisable_info
+
+ if serialisable_context_data is None:
+
+ self.context_data = serialisable_context_data
+
+ else:
+
+ self.context_data = serialisable_context_data.decode( 'hex' )
+
+
+
+HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_CONTEXT ] = NetworkContext
+
+GLOBAL_NETWORK_CONTEXT = NetworkContext( CC.NETWORK_CONTEXT_GLOBAL )
class NetworkEngine( object ):
MAX_JOBS = 10 # turn this into an option
- def __init__( self, controller ):
+ def __init__( self, controller, bandwidth_manager, session_manager, login_manager ):
- self._controller = controller
+ self.controller = controller
+
+ self.bandwidth_manager = bandwidth_manager
+ self.session_manager = session_manager
+ self.login_manager = login_manager
+
+ self.bandwidth_manager.engine = self
+ self.session_manager.engine = self
+ self.login_manager.engine = self
self._lock = threading.Lock()
@@ -1208,24 +1307,44 @@ class NetworkEngine( object ):
self._jobs_ready_to_start = []
self._jobs_downloading = []
+ self._is_running = False
+ self._is_shutdown = False
self._local_shutdown = False
- self._controller.CallToThread( self.MainLoop )
-
def AddJob( self, job ):
with self._lock:
+ job.engine = self
+
self._jobs_bandwidth_throttled.append( job )
self._new_work_to_do.set()
+ def IsRunning( self ):
+
+ with self._lock:
+
+ return self._is_running
+
+
+
+ def IsShutdown( self ):
+
+ with self._lock:
+
+ return self._is_shutdown
+
+
+
def MainLoop( self ):
- while not ( self._local_shutdown or self._controller.ModelIsShutdown() ):
+ self._is_running = True
+
+ while not ( self._local_shutdown or self.controller.ModelIsShutdown() ):
def ProcessBandwidthJob( job ):
@@ -1271,7 +1390,7 @@ class NetworkEngine( object ):
login_process = job.GenerateLoginProcess()
- self._controller.CallToThread( login_process.Start )
+ self.controller.CallToThread( login_process.Start )
self._current_login_process = login_process
@@ -1320,7 +1439,7 @@ class NetworkEngine( object ):
elif len( self._jobs_downloading ) < self.MAX_JOBS:
- self._controller.CallToThread( job.Start )
+ self.controller.CallToThread( job.Start )
self._jobs_downloading.append( job )
@@ -1362,15 +1481,28 @@ class NetworkEngine( object ):
self._new_work_to_do.clear()
+ self._is_running = False
+
+ self._is_shutdown = True
+
+
+ def Start( self ):
+
+ self.controller.CallToThread( self.MainLoop )
+
def Shutdown( self ):
self._local_shutdown = True
+ self._new_work_to_do.set()
+
class NetworkJob( object ):
- def __init__( self, method, url, body = None, referral_url = None, temp_path = None ):
+ def __init__( self, method, url, body = None, referral_url = None, temp_path = None, for_login = False ):
+
+ self.engine = None
self._lock = threading.Lock()
@@ -1379,6 +1511,7 @@ class NetworkJob( object ):
self._body = body
self._referral_url = referral_url
self._temp_path = temp_path
+ self._for_login = for_login
self._bandwidth_tracker = HydrusNetworking.BandwidthTracker()
@@ -1392,7 +1525,7 @@ class NetworkJob( object ):
self._is_done = False
self._is_cancelled = False
- self._bandwidth_override = False
+ self._bandwidth_manual_override = False
self._status_code = None
@@ -1400,30 +1533,33 @@ class NetworkJob( object ):
self._num_bytes_read = 0
self._num_bytes_to_read = None
-
- def _BandwidthOK( self ):
-
- raise NotImplementedError()
+ self._network_contexts = self._GenerateNetworkContexts()
- def _CanLogin( self ):
+ def _GenerateNetworkContexts( self ):
- raise NotImplementedError()
+ network_contexts = []
-
- def _GenerateLoginProcess( self ):
+ network_contexts.append( GLOBAL_NETWORK_CONTEXT )
- raise NotImplementedError()
+ domain = ConvertURLIntoDomain( self._url )
+ domains = ConvertDomainIntoAllApplicableDomains( domain )
+
+ network_contexts.extend( ( NetworkContext( CC.NETWORK_CONTEXT_DOMAIN, domain ) for domain in domains ) )
+
+ return network_contexts
def _GetSession( self ):
- raise NotImplementedError()
+ session_network_context = self._GetSessionNetworkContext()
+
+ return self.engine.session_manager.GetSession( session_network_context )
- def _ImmediateBandwidthOK( self ):
+ def _GetSessionNetworkContext( self ):
- raise NotImplementedError()
+ return self._network_contexts[-1]
def _IsCancelled( self ):
@@ -1433,7 +1569,7 @@ class NetworkJob( object ):
return True
- if HG.client_controller.ModelIsShutdown():
+ if self.engine.controller.ModelIsShutdown():
return True
@@ -1441,9 +1577,36 @@ class NetworkJob( object ):
return False
- def _NeedsLogin( self ):
+ def _IsDone( self ):
- raise NotImplementedError()
+ if self._is_done:
+
+ return True
+
+
+ if self.engine.controller.ModelIsShutdown():
+
+ return True
+
+
+ return False
+
+
+ def _ObeysBandwidth( self ):
+
+ return self._bandwidth_manual_override or self._for_login
+
+
+ def _OngoingBandwidthOK( self ):
+
+ if self._ObeysBandwidth():
+
+ return self.engine.bandwidth_manager.CanContinue( self._network_contexts )
+
+ else:
+
+ return True
+
def _ReadResponse( self, response, stream_dest ):
@@ -1455,7 +1618,7 @@ class NetworkJob( object ):
try:
- for chunk in response.iter_content( chunk_size = 8192 ):
+ for chunk in response.iter_content( chunk_size = 65536 ):
if self._IsCancelled():
@@ -1469,7 +1632,7 @@ class NetworkJob( object ):
self._num_bytes_read += chunk_length
self._ReportDataUsed( chunk_length )
- self._WaitOnImmediateBandwidth()
+ self._WaitOnOngoingBandwidth()
finally:
@@ -1487,11 +1650,15 @@ class NetworkJob( object ):
self._bandwidth_tracker.ReportDataUsed( num_bytes )
+ self.engine.bandwidth_manager.ReportDataUsed( self._network_contexts, num_bytes )
+
def _ReportRequestUsed( self ):
self._bandwidth_tracker.ReportRequestUsed()
+ self.engine.bandwidth_manager.ReportRequestUsed( self._network_contexts )
+
def _SetCancelled( self ):
@@ -1514,9 +1681,9 @@ class NetworkJob( object ):
self._is_done = True
- def _WaitOnImmediateBandwidth( self ):
+ def _WaitOnOngoingBandwidth( self ):
- while not self._ImmediateBandwidthOK() and not self._IsCancelled():
+ while not self._OngoingBandwidthOK() and not self._IsCancelled():
time.sleep( 0.5 )
@@ -1526,13 +1693,13 @@ class NetworkJob( object ):
with self._lock:
- if self._bandwidth_override:
+ if self._ObeysBandwidth:
- return True
+ return self.engine.bandwidth_manager.CanStart( self._network_contexts )
else:
- return self._BandwidthOK()
+ return True
@@ -1547,11 +1714,33 @@ class NetworkJob( object ):
+ def CanLogin( self ):
+
+ with self._lock:
+
+ if self._for_login:
+
+ raise Exception( 'Login jobs should not be asked if they can login!' )
+
+ else:
+
+ return self.engine.login_manager.CanLogin( self._network_contexts )
+
+
+
+
def GenerateLoginProcess( self ):
with self._lock:
- return self._GenerateLoginProcess()
+ if self._for_login:
+
+ raise Exception( 'Login jobs should not be asked to generate login processes!' )
+
+ else:
+
+ return self.engine.login_manager.GenerateLoginProcess( self._network_contexts )
+
@@ -1601,7 +1790,7 @@ class NetworkJob( object ):
with self._lock:
- return HydrusData.TimeHasPassed( self._wake_time )
+ return not HydrusData.TimeHasPassed( self._wake_time )
@@ -1617,7 +1806,7 @@ class NetworkJob( object ):
with self._lock:
- return self._is_done
+ return self._IsDone()
@@ -1625,7 +1814,14 @@ class NetworkJob( object ):
with self._lock:
- self._NeedsLogin()
+ if self._for_login:
+
+ return False
+
+ else:
+
+ return self.engine.login_manager.NeedsLogin( self._network_contexts )
+
@@ -1633,7 +1829,7 @@ class NetworkJob( object ):
with self._lock:
- self._bandwidth_override = True
+ self._bandwidth_manual_override = True
@@ -1754,184 +1950,227 @@ class NetworkJob( object ):
-class NetworkJobWeb( NetworkJob ):
+class NetworkJobDownloader( NetworkJob ):
- def _BandwidthOK( self ):
+ def __init__( self, downloader_key, method, url, body = None, temp_path = None ):
- bandwidth_manager = HG.client_controller.GetBandwidthManager()
+ self._downloader_key = downloader_key
- return bandwidth_manager.CanStartURL( self._url )
+ NetworkJob.__init__( self, method, url, body, temp_path = temp_path )
- def _CanLogin( self ):
+ def _GenerateNetworkContexts( self ):
- # ask login engine if it is possible to login at this time (i.e. if our login details seem to be valid--a bad login will invalidate the form/details until the user can re-verify, hence stopping all bad requests from spamming a changed login form)
+ network_contexts = NetworkJob._GenerateNetworkContexts( self )
- pass
+ network_contexts.append( NetworkContext( CC.NETWORK_CONTEXT_DOWNLOADER, self._downloader_key ) )
+
+ return network_contexts
- def _GenerateLoginProcess( self ):
+class NetworkJobSubscription( NetworkJobDownloader ):
+
+ def __init__( self, subscription_key, downloader_key, method, url, body = None, temp_path = None ):
- pass
+ self._subscription_key = subscription_key
- # talk to login engine, figure out an object to handle this that will follow the script and report status back to the network engine
+ NetworkJobDownloader.__init__( self, downloader_key, method, url, body, temp_path = temp_path )
- def _GetSession( self ):
+ def _GenerateNetworkContexts( self ):
- pass # fetch the regular session from the sessionmanager
+ network_contexts = NetworkJob._GenerateNetworkContexts( self )
+
+ network_contexts.append( NetworkContext( CC.NETWORK_CONTEXT_SUBSCRIPTION, self._subscription_key ) )
+
+ return network_contexts
- def _ImmediateBandwidthOK( self ):
+ def _GetSessionNetworkContext( self ):
- bandwidth_manager = HG.client_controller.GetBandwidthManager()
-
- return bandwidth_manager.CanContinueURL( self._url )
-
-
- def _NeedsLogin( self ):
-
- # consult login engine, ask if I need login (it will consult its records and the current session)
-
- pass
-
-
- def _ReportDataUsed( self, num_bytes ):
-
- NetworkJob._ReportDataUsed( self, num_bytes )
-
- bandwidth_manager = HG.client_controller.GetBandwidthManager()
-
- bandwidth_manager.ReportDataUsedURL( self._url, num_bytes )
-
-
- def _ReportRequestUsed( self ):
-
- NetworkJob._ReportRequestUsed( self )
-
- bandwidth_manager = HG.client_controller.GetBandwidthManager()
-
- bandwidth_manager.ReportRequestUsedURL( self._url )
-
-
-class NetworkJobWebLogin( NetworkJobWeb ):
-
- def _BandwidthOK( self ):
-
- return True
-
-
- def _ImmediateBandwidthOK( self ):
-
- return True
-
-
- def _NeedsLogin( self ):
-
- return False
+ return self._network_contexts[-2] # the downloader one
class NetworkJobHydrus( NetworkJob ):
def __init__( self, service_key, method, url, body = None, temp_path = None ):
- NetworkJob.__init__( self, method, url, body, temp_path = temp_path )
-
self._service_key = service_key
+ NetworkJob.__init__( self, method, url, body, temp_path = temp_path )
+
- def _BandwidthOK( self ):
+ def _GenerateNetworkContexts( self ):
- bandwidth_manager = HG.client_controller.GetBandwidthManager()
+ network_contexts = NetworkJob._GenerateNetworkContexts( self )
- if not bandwidth_manager.CanStartGlobally():
+ network_contexts.append( NetworkContext( CC.NETWORK_CONTEXT_HYDRUS, self._service_key ) )
+
+ return network_contexts
+
+
+class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
+
+ SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_LOGIN_MANAGER
+ SERIALISABLE_VERSION = 1
+
+ def __init__( self ):
+
+ HydrusSerialisable.SerialisableBase.__init__( self )
+
+ self.engine = None
+
+ self._lock = threading.Lock()
+
+ # for every loginnable network_context, we need
+ # a login script
+ # hydrus script is different, obvs
+ # start with this
+ # domain login scripts are complicated
+ # say this is notimplemented yet, then convert hf and pixiv over
+ # current login status
+ # current login expiry
+
+ # essentially, we need to answer:
+ # needslogin
+ # canlogin
+ # dologin
+
+ self._network_contexts_to_sessions = {}
+
+
+ def _GenerateSession( self, network_context ):
+
+ session = requests.Session()
+
+ session.headers.update( { 'User-Agent', 'hydrus/' + str( HC.NETWORK_VERSION ) } )
+
+ if network_context.context_type == CC.NETWORK_CONTEXT_HYDRUS:
- return False
+ session.verify = False
- else:
+
+ return session
+
+
+ def _GetSerialisableInfo( self ):
+
+ serialisable_network_contexts_to_sessions = [ ( network_context.GetSerialisableTuple(), cPickle.dumps( session ) ) for ( network_context, session ) in self._network_contexts_to_sessions.items() ]
+
+ return serialisable_network_contexts_to_sessions
+
+
+ def _InitialiseFromSerialisableInfo( self, serialisable_info ):
+
+ serialisable_network_contexts_to_sessions = serialisable_info
+
+ for ( serialisable_network_context, pickled_session ) in serialisable_network_contexts_to_sessions:
- service = HG.client_controller.GetServicesManager().GetService( self._service_key )
+ network_context = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_network_context )
+ session = cPickle.loads( pickled_session )
- return service.BandwidthOK()
+ self._network_contexts_to_sessions[ network_context ] = session
- def _CanLogin( self ):
+ def ClearSession( self, network_context ):
- # ask service if account is valid
-
- pass
+ with self._lock:
+
+ if network_context in self._network_contexts_to_sessions:
+
+ del self._network_contexts_to_sessions[ network_context ]
+
+
- def _GenerateLoginProcess( self ):
+ def GetSession( self, network_context ):
- pass
-
- # talk to service, figure out a login-process compatible object to handle the session gen
+ with self._lock:
+
+ if network_context not in self._network_contexts_to_sessions:
+
+ self._network_contexts_to_sessions[ network_context ] = self._GenerateSession( network_context )
+
+
+ return self._network_contexts_to_sessions[ network_context ]
+
- def _GetSession( self ):
+HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_LOGIN_MANAGER ] = NetworkLoginManager
+
+class NetworkSessionManager( HydrusSerialisable.SerialisableBase ):
+
+ SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER
+ SERIALISABLE_VERSION = 1
+
+ def __init__( self ):
- pass # fetch the hydrus (ssl verify=False) session, which should have the keys as cookies, right?
- # this will ultimately be a job for the login engine step, earlier
+ HydrusSerialisable.SerialisableBase.__init__( self )
+
+ self.engine = None
+
+ self._lock = threading.Lock()
+
+ self._network_contexts_to_sessions = {}
- def _ImmediateBandwidthOK( self ):
+ def _GenerateSession( self, network_context ):
- bandwidth_manager = HG.client_controller.GetBandwidthManager()
+ session = requests.Session()
- service = HG.client_controller.GetServicesManager().GetService( self._service_key )
+ session.headers.update( { 'User-Agent', 'hydrus/' + str( HC.NETWORK_VERSION ) } )
- return bandwidth_manager.CanContinueGlobally() and service.CanContinue()
+ if network_context.context_type == CC.NETWORK_CONTEXT_HYDRUS:
+
+ session.verify = False
+
+
+ return session
- def _NeedsLogin( self ):
+ def _GetSerialisableInfo( self ):
- # consult service, ask if I need a session key
+ serialisable_network_contexts_to_sessions = [ ( network_context.GetSerialisableTuple(), cPickle.dumps( session ) ) for ( network_context, session ) in self._network_contexts_to_sessions.items() ]
- pass
+ return serialisable_network_contexts_to_sessions
- def _ReportDataUsed( self, num_bytes ):
+ def _InitialiseFromSerialisableInfo( self, serialisable_info ):
- NetworkJob._ReportDataUsed( self, num_bytes )
+ serialisable_network_contexts_to_sessions = serialisable_info
- bandwidth_manager = HG.client_controller.GetBandwidthManager()
-
- bandwidth_manager.ReportDataUsedGlobally( num_bytes )
-
- service = HG.client_controller.GetServicesManager().GetService( self._service_key )
-
- return service.ReportDataUsed( num_bytes )
+ for ( serialisable_network_context, pickled_session ) in serialisable_network_contexts_to_sessions:
+
+ network_context = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_network_context )
+ session = cPickle.loads( pickled_session )
+
+ self._network_contexts_to_sessions[ network_context ] = session
+
- def _ReportRequestUsed( self ):
+ def ClearSession( self, network_context ):
- NetworkJob._ReportRequestUsed( self )
-
- bandwidth_manager = HG.client_controller.GetBandwidthManager()
-
- bandwidth_manager.ReportRequestUsedGlobally()
-
- service = HG.client_controller.GetServicesManager().GetService( self._service_key )
-
- return service.ReportRequestUsed()
+ with self._lock:
+
+ if network_context in self._network_contexts_to_sessions:
+
+ del self._network_contexts_to_sessions[ network_context ]
+
+
-class NetworkJobHydrusLogin( NetworkJobHydrus ):
-
- def _BandwidthOK( self ):
+ def GetSession( self, network_context ):
- return True
-
-
- def _ImmediateBandwidthOK( self ):
-
- return True
-
-
- def _NeedsLogin( self ):
-
- return False
+ with self._lock:
+
+ if network_context not in self._network_contexts_to_sessions:
+
+ self._network_contexts_to_sessions[ network_context ] = self._GenerateSession( network_context )
+
+
+ return self._network_contexts_to_sessions[ network_context ]
+
+HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER ] = NetworkSessionManager
diff --git a/include/ClientParsing.py b/include/ClientParsing.py
index ba75f4ac..cd68ddad 100644
--- a/include/ClientParsing.py
+++ b/include/ClientParsing.py
@@ -744,7 +744,7 @@ class ParseRootFileLookup( HydrusSerialisable.SerialisableBaseNamed ):
hash = media.GetHash()
mime = media.GetMime()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
try:
diff --git a/include/ClientRatings.py b/include/ClientRatings.py
index 66ca3881..de126701 100644
--- a/include/ClientRatings.py
+++ b/include/ClientRatings.py
@@ -188,7 +188,7 @@ def GetNumericalStateFromMedia( media, service_key ):
def GetNumericalWidth( service_key ):
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
num_stars = service.GetNumStars()
@@ -196,7 +196,7 @@ def GetNumericalWidth( service_key ):
def GetPenAndBrushColours( service_key, rating_state ):
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
colour = service.GetColour( rating_state )
@@ -209,7 +209,7 @@ def GetPenAndBrushColours( service_key, rating_state ):
def GetShape( service_key ):
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
shape = service.GetShape()
@@ -217,7 +217,7 @@ def GetShape( service_key ):
def GetStars( service_key, rating_state, rating ):
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
allow_zero = service.AllowZero()
diff --git a/include/ClientRendering.py b/include/ClientRendering.py
index ee708c35..48120d23 100644
--- a/include/ClientRendering.py
+++ b/include/ClientRendering.py
@@ -60,7 +60,7 @@ class ImageRenderer( object ):
hash = self._media.GetHash()
mime = self._media.GetMime()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
self._path = client_files_manager.GetFilePath( hash, mime )
@@ -158,7 +158,7 @@ class RasterContainer( object ):
hash = self._media.GetHash()
mime = self._media.GetMime()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
self._path = client_files_manager.GetFilePath( hash, mime )
@@ -329,7 +329,7 @@ class RasterContainerVideo( RasterContainer ):
duration = self._media.GetDuration()
num_frames = self._media.GetNumFrames()
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
if self._media.GetMime() == HC.IMAGE_GIF:
@@ -441,8 +441,14 @@ class RasterContainerVideo( RasterContainer ):
def GetDuration( self, index ):
- if self._media.GetMime() == HC.IMAGE_GIF: return self._durations[ index ]
- else: return self._average_frame_duration
+ if self._media.GetMime() == HC.IMAGE_GIF:
+
+ return self._durations[ index ]
+
+ else:
+
+ return self._average_frame_duration
+
def GetFrame( self, index ):
@@ -546,8 +552,14 @@ class RasterContainerVideo( RasterContainer ):
def GetTotalDuration( self ):
- if self._media.GetMime() == HC.IMAGE_GIF: return sum( self._durations )
- else: return self._average_frame_duration * self.GetNumFrames()
+ if self._media.GetMime() == HC.IMAGE_GIF:
+
+ return sum( self._durations )
+
+ else:
+
+ return self._average_frame_duration * self.GetNumFrames()
+
def HasFrame( self, index ):
diff --git a/include/ClientSearch.py b/include/ClientSearch.py
index 16433aac..ebcef79e 100644
--- a/include/ClientSearch.py
+++ b/include/ClientSearch.py
@@ -296,7 +296,7 @@ class FileSearchContext( HydrusSerialisable.SerialisableBase ):
self._file_service_key = file_service_key.decode( 'hex' )
self._tag_service_key = tag_service_key.decode( 'hex' )
- services_manager = HG.client_controller.GetServicesManager()
+ services_manager = HG.client_controller.services_manager
if not services_manager.ServiceExists( self._file_service_key ):
@@ -1101,7 +1101,7 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
( operator, value, service_key ) = self._value
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
service_type = service.GetServiceType()
@@ -1176,7 +1176,7 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
if current_or_pending == HC.CONTENT_STATUS_PENDING: base += u' pending to '
else: base += u' currently in '
- service = HG.client_controller.GetServicesManager().GetService( service_key )
+ service = HG.client_controller.services_manager.GetService( service_key )
base += service.GetName()
diff --git a/include/ClientServices.py b/include/ClientServices.py
index 9afd1e03..58df0f14 100644
--- a/include/ClientServices.py
+++ b/include/ClientServices.py
@@ -1408,7 +1408,7 @@ class ServiceRepository( ServiceRestricted ):
if num_to_do > 0:
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
job_key = ClientThreading.JobKey( cancellable = True, stop_time = stop_time )
@@ -1752,7 +1752,7 @@ class ServiceIPFS( ServiceRemote ):
url = api_base_url + 'add'
- client_files_manager = HG.client_controller.GetClientFilesManager()
+ client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFilePath( hash, mime )
diff --git a/include/ClientTags.py b/include/ClientTags.py
index 9e547b9b..91bd2557 100644
--- a/include/ClientTags.py
+++ b/include/ClientTags.py
@@ -78,7 +78,7 @@ def ImportFromHTA( parent, hta_path, tag_service_key, hashes ):
del hta
- service = HG.client_controller.GetServicesManager().GetService( tag_service_key )
+ service = HG.client_controller.services_manager.GetService( tag_service_key )
service_type = service.GetServiceType()
@@ -166,7 +166,7 @@ def ImportFromHTA( parent, hta_path, tag_service_key, hashes ):
text += os.linesep.join( HydrusData.ConvertUglyNamespacesToPrettyStrings( namespaces ) )
text += os.linesep * 2
- file_service = HG.client_controller.GetServicesManager().GetService( file_service_key )
+ file_service = HG.client_controller.services_manager.GetService( file_service_key )
text += 'For '
diff --git a/include/HydrusConstants.py b/include/HydrusConstants.py
index d0632725..1d4f9f85 100755
--- a/include/HydrusConstants.py
+++ b/include/HydrusConstants.py
@@ -49,7 +49,7 @@ options = {}
# Misc
NETWORK_VERSION = 18
-SOFTWARE_VERSION = 261
+SOFTWARE_VERSION = 262
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )
@@ -418,7 +418,7 @@ UNDETERMINED_WM = 19
VIDEO_MKV = 20
VIDEO_WEBM = 21
APPLICATION_JSON = 22
-VIDEO_APNG = 23
+IMAGE_APNG = 23
UNDETERMINED_PNG = 24
VIDEO_MPEG = 25
VIDEO_MOV = 26
@@ -429,16 +429,16 @@ TEXT_PLAIN = 30
APPLICATION_OCTET_STREAM = 100
APPLICATION_UNKNOWN = 101
-ALLOWED_MIMES = ( IMAGE_JPEG, IMAGE_PNG, IMAGE_GIF, IMAGE_BMP, APPLICATION_FLASH, VIDEO_AVI, VIDEO_FLV, VIDEO_MOV, VIDEO_MP4, VIDEO_MKV, VIDEO_WEBM, VIDEO_MPEG, APPLICATION_PDF, AUDIO_MP3, AUDIO_OGG, AUDIO_FLAC, AUDIO_WMA, VIDEO_WMV, APPLICATION_HYDRUS_UPDATE_CONTENT, APPLICATION_HYDRUS_UPDATE_DEFINITIONS )
-SEARCHABLE_MIMES = ( IMAGE_JPEG, IMAGE_PNG, IMAGE_GIF, APPLICATION_FLASH, VIDEO_AVI, VIDEO_FLV, VIDEO_MOV, VIDEO_MP4, VIDEO_MKV, VIDEO_WEBM, VIDEO_MPEG, APPLICATION_PDF, AUDIO_MP3, AUDIO_OGG, AUDIO_FLAC, AUDIO_WMA, VIDEO_WMV )
+ALLOWED_MIMES = ( IMAGE_JPEG, IMAGE_PNG, IMAGE_APNG, IMAGE_GIF, IMAGE_BMP, APPLICATION_FLASH, VIDEO_AVI, VIDEO_FLV, VIDEO_MOV, VIDEO_MP4, VIDEO_MKV, VIDEO_WEBM, VIDEO_MPEG, APPLICATION_PDF, AUDIO_MP3, AUDIO_OGG, AUDIO_FLAC, AUDIO_WMA, VIDEO_WMV, APPLICATION_HYDRUS_UPDATE_CONTENT, APPLICATION_HYDRUS_UPDATE_DEFINITIONS )
+SEARCHABLE_MIMES = ( IMAGE_JPEG, IMAGE_PNG, IMAGE_APNG, IMAGE_GIF, APPLICATION_FLASH, VIDEO_AVI, VIDEO_FLV, VIDEO_MOV, VIDEO_MP4, VIDEO_MKV, VIDEO_WEBM, VIDEO_MPEG, APPLICATION_PDF, AUDIO_MP3, AUDIO_OGG, AUDIO_FLAC, AUDIO_WMA, VIDEO_WMV )
-IMAGES = ( IMAGE_JPEG, IMAGE_PNG, IMAGE_GIF, IMAGE_BMP )
+IMAGES = ( IMAGE_JPEG, IMAGE_PNG, IMAGE_APNG, IMAGE_GIF, IMAGE_BMP )
AUDIO = ( AUDIO_MP3, AUDIO_OGG, AUDIO_FLAC, AUDIO_WMA )
VIDEO = ( VIDEO_AVI, VIDEO_FLV, VIDEO_MOV, VIDEO_MP4, VIDEO_WMV, VIDEO_MKV, VIDEO_WEBM, VIDEO_MPEG )
-NATIVE_VIDEO = ( VIDEO_AVI, VIDEO_FLV, VIDEO_MOV, VIDEO_MP4, VIDEO_WMV, VIDEO_MKV, VIDEO_WEBM, VIDEO_MPEG )
+NATIVE_VIDEO = ( IMAGE_APNG, VIDEO_AVI, VIDEO_FLV, VIDEO_MOV, VIDEO_MP4, VIDEO_WMV, VIDEO_MKV, VIDEO_WEBM, VIDEO_MPEG )
APPLICATIONS = ( APPLICATION_FLASH, APPLICATION_PDF, APPLICATION_ZIP )
@@ -446,7 +446,7 @@ NOISY_MIMES = tuple( [ APPLICATION_FLASH ] + list( AUDIO ) + list( VIDEO ) )
ARCHIVES = ( APPLICATION_ZIP, APPLICATION_HYDRUS_ENCRYPTED_ZIP )
-MIMES_WITH_THUMBNAILS = ( APPLICATION_FLASH, IMAGE_JPEG, IMAGE_PNG, IMAGE_GIF, IMAGE_BMP, VIDEO_AVI, VIDEO_FLV, VIDEO_MOV, VIDEO_MP4, VIDEO_WMV, VIDEO_MKV, VIDEO_WEBM, VIDEO_MPEG )
+MIMES_WITH_THUMBNAILS = ( APPLICATION_FLASH, IMAGE_JPEG, IMAGE_PNG, IMAGE_APNG, IMAGE_GIF, IMAGE_BMP, VIDEO_AVI, VIDEO_FLV, VIDEO_MOV, VIDEO_MP4, VIDEO_WMV, VIDEO_MKV, VIDEO_WEBM, VIDEO_MPEG )
MIMES_WE_CAN_PHASH = ( IMAGE_JPEG, IMAGE_PNG )
@@ -460,6 +460,7 @@ mime_enum_lookup[ 'image/jpeg' ] = IMAGE_JPEG
mime_enum_lookup[ 'image/jpg' ] = IMAGE_JPEG
mime_enum_lookup[ 'image/x-png' ] = IMAGE_PNG
mime_enum_lookup[ 'image/png' ] = IMAGE_PNG
+mime_enum_lookup[ 'image/apng' ] = IMAGE_APNG
mime_enum_lookup[ 'image/gif' ] = IMAGE_GIF
mime_enum_lookup[ 'image/bmp' ] = IMAGE_BMP
mime_enum_lookup[ 'image' ] = IMAGES
@@ -481,7 +482,6 @@ mime_enum_lookup[ 'audio/flac' ] = AUDIO_FLAC
mime_enum_lookup[ 'audio/x-ms-wma' ] = AUDIO_WMA
mime_enum_lookup[ 'text/html' ] = TEXT_HTML
mime_enum_lookup[ 'text/plain' ] = TEXT_PLAIN
-mime_enum_lookup[ 'video/png' ] = VIDEO_APNG
mime_enum_lookup[ 'video/x-msvideo' ] = VIDEO_AVI
mime_enum_lookup[ 'video/x-flv' ] = VIDEO_FLV
mime_enum_lookup[ 'video/quicktime' ] = VIDEO_MOV
@@ -498,6 +498,7 @@ mime_string_lookup = {}
mime_string_lookup[ APPLICATION_HYDRUS_CLIENT_COLLECTION ] = 'collection'
mime_string_lookup[ IMAGE_JPEG ] = 'image/jpg'
mime_string_lookup[ IMAGE_PNG ] = 'image/png'
+mime_string_lookup[ IMAGE_APNG ] = 'image/apng'
mime_string_lookup[ IMAGE_GIF ] = 'image/gif'
mime_string_lookup[ IMAGE_BMP ] = 'image/bmp'
mime_string_lookup[ IMAGES ] = 'image'
@@ -519,7 +520,6 @@ mime_string_lookup[ AUDIO_WMA ] = 'audio/x-ms-wma'
mime_string_lookup[ AUDIO ] = 'audio'
mime_string_lookup[ TEXT_HTML ] = 'text/html'
mime_string_lookup[ TEXT_PLAIN ] = 'text/plain'
-mime_string_lookup[ VIDEO_APNG ] = 'video/png'
mime_string_lookup[ VIDEO_AVI ] = 'video/x-msvideo'
mime_string_lookup[ VIDEO_FLV ] = 'video/x-flv'
mime_string_lookup[ VIDEO_MOV ] = 'video/quicktime'
@@ -537,6 +537,7 @@ mime_ext_lookup = {}
mime_ext_lookup[ APPLICATION_HYDRUS_CLIENT_COLLECTION ] = '.collection'
mime_ext_lookup[ IMAGE_JPEG ] = '.jpg'
mime_ext_lookup[ IMAGE_PNG ] = '.png'
+mime_ext_lookup[ IMAGE_APNG ] = '.png'
mime_ext_lookup[ IMAGE_GIF ] = '.gif'
mime_ext_lookup[ IMAGE_BMP ] = '.bmp'
mime_ext_lookup[ IMAGE_ICON ] = '.ico'
@@ -555,7 +556,6 @@ mime_ext_lookup[ AUDIO_FLAC ] = '.flac'
mime_ext_lookup[ AUDIO_WMA ] = '.wma'
mime_ext_lookup[ TEXT_HTML ] = '.html'
mime_ext_lookup[ TEXT_PLAIN ] = '.txt'
-mime_ext_lookup[ VIDEO_APNG ] = '.png'
mime_ext_lookup[ VIDEO_AVI ] = '.avi'
mime_ext_lookup[ VIDEO_FLV ] = '.flv'
mime_ext_lookup[ VIDEO_MOV ] = '.mov'
diff --git a/include/HydrusFileHandling.py b/include/HydrusFileHandling.py
index b6fda3bc..73070622 100644
--- a/include/HydrusFileHandling.py
+++ b/include/HydrusFileHandling.py
@@ -70,60 +70,77 @@ def GenerateThumbnail( path, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS ):
mime = GetMime( path )
- f = cStringIO.StringIO()
-
- if mime in HC.IMAGES:
+ if mime in ( HC.IMAGE_JPEG, HC.IMAGE_PNG, HC.IMAGE_GIF ):
- pil_image = HydrusImageHandling.GeneratePILImage( path )
-
- SaveThumbnailToStream( pil_image, dimensions, f )
-
- elif mime == HC.APPLICATION_FLASH:
-
- ( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
-
- try:
-
- HydrusFlashHandling.RenderPageToFile( path, temp_path, 1 )
-
- pil_image = HydrusImageHandling.GeneratePILImage( temp_path )
-
- SaveThumbnailToStream( pil_image, dimensions, f )
-
- except:
-
- flash_default_path = os.path.join( HC.STATIC_DIR, 'flash.png' )
-
- pil_image = HydrusImageHandling.GeneratePILImage( flash_default_path )
-
- SaveThumbnailToStream( pil_image, dimensions, f )
-
- finally:
-
- del pil_image
-
- HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
-
+ thumbnail = GenerateThumbnailFromStaticImage( path, dimensions )
else:
- ( size, mime, width, height, duration, num_frames, num_words ) = GetFileInfo( path )
+ f = cStringIO.StringIO()
- cropped_dimensions = HydrusImageHandling.GetThumbnailResolution( ( width, height ), dimensions )
-
- renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, cropped_dimensions )
-
- numpy_image = renderer.read_frame()
-
- if numpy_image is None:
+ if mime == HC.APPLICATION_FLASH:
- raise Exception( 'Could not create a thumbnail from that video!' )
+ ( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
+
+ try:
+
+ HydrusFlashHandling.RenderPageToFile( path, temp_path, 1 )
+
+ pil_image = HydrusImageHandling.GeneratePILImage( temp_path )
+
+ SaveThumbnailToStream( pil_image, dimensions, f )
+
+ except:
+
+ flash_default_path = os.path.join( HC.STATIC_DIR, 'flash.png' )
+
+ pil_image = HydrusImageHandling.GeneratePILImage( flash_default_path )
+
+ SaveThumbnailToStream( pil_image, dimensions, f )
+
+ finally:
+
+ del pil_image
+
+ HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
+
+
+ else:
+
+ ( size, mime, width, height, duration, num_frames, num_words ) = GetFileInfo( path )
+
+ cropped_dimensions = HydrusImageHandling.GetThumbnailResolution( ( width, height ), dimensions )
+
+ renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, cropped_dimensions )
+
+ numpy_image = renderer.read_frame()
+
+ if numpy_image is None:
+
+ raise Exception( 'Could not create a thumbnail from that video!' )
+
+
+ pil_image = HydrusImageHandling.GeneratePILImageFromNumpyImage( numpy_image )
+
+ SaveThumbnailToStream( pil_image, dimensions, f )
- pil_image = HydrusImageHandling.GeneratePILImageFromNumpyImage( numpy_image )
+ f.seek( 0 )
- SaveThumbnailToStream( pil_image, dimensions, f )
+ thumbnail = f.read()
+ f.close()
+
+
+ return thumbnail
+
+def GenerateThumbnailFromStaticImage( path, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS ):
+
+ f = cStringIO.StringIO()
+
+ pil_image = HydrusImageHandling.GeneratePILImage( path )
+
+ SaveThumbnailToStream( pil_image, dimensions, f )
f.seek( 0 )
@@ -174,7 +191,7 @@ def GetFileInfo( path ):
num_frames = None
num_words = None
- if mime in HC.IMAGES:
+ if mime in ( HC.IMAGE_JPEG, HC.IMAGE_PNG, HC.IMAGE_GIF ):
( ( width, height ), duration, num_frames ) = HydrusImageHandling.GetImageProperties( path )
@@ -182,15 +199,30 @@ def GetFileInfo( path ):
( ( width, height ), duration, num_frames ) = HydrusFlashHandling.GetFlashProperties( path )
- elif mime in ( HC.VIDEO_AVI, HC.VIDEO_FLV, HC.VIDEO_WMV, HC.VIDEO_MOV, HC.VIDEO_MP4, HC.VIDEO_MKV, HC.VIDEO_WEBM, HC.VIDEO_MPEG ):
+ elif mime in ( HC.IMAGE_APNG, HC.VIDEO_AVI, HC.VIDEO_FLV, HC.VIDEO_WMV, HC.VIDEO_MOV, HC.VIDEO_MP4, HC.VIDEO_MKV, HC.VIDEO_WEBM, HC.VIDEO_MPEG ):
( ( width, height ), duration, num_frames ) = HydrusVideoHandling.GetFFMPEGVideoProperties( path )
- elif mime == HC.APPLICATION_PDF: num_words = HydrusDocumentHandling.GetPDFNumWords( path )
- elif mime == HC.AUDIO_MP3: duration = HydrusAudioHandling.GetMP3Duration( path )
- elif mime == HC.AUDIO_OGG: duration = HydrusAudioHandling.GetOGGVorbisDuration( path )
- elif mime == HC.AUDIO_FLAC: duration = HydrusAudioHandling.GetFLACDuration( path )
- elif mime == HC.AUDIO_WMA: duration = HydrusAudioHandling.GetWMADuration( path )
+ elif mime == HC.APPLICATION_PDF:
+
+ num_words = HydrusDocumentHandling.GetPDFNumWords( path )
+
+ elif mime == HC.AUDIO_MP3:
+
+ duration = HydrusAudioHandling.GetMP3Duration( path )
+
+ elif mime == HC.AUDIO_OGG:
+
+ duration = HydrusAudioHandling.GetOGGVorbisDuration( path )
+
+ elif mime == HC.AUDIO_FLAC:
+
+ duration = HydrusAudioHandling.GetFLACDuration( path )
+
+ elif mime == HC.AUDIO_WMA:
+
+ duration = HydrusAudioHandling.GetWMADuration( path )
+
return ( size, mime, width, height, duration, num_frames, num_words )
@@ -231,18 +263,14 @@ def GetMime( path ):
elif mime == HC.UNDETERMINED_PNG:
- return HC.IMAGE_PNG
-
- # atm (Feb 2016), ffmpeg doesn't report duration for apngs, so can't do this just yet.
- #
- #if HydrusVideoHandling.HasVideoStream( path ):
- #
- # return HC.VIDEO_APNG
- #
- #else:
- #
- # return HC.IMAGE_PNG
- #
+ if HydrusVideoHandling.HasVideoStream( path ):
+
+ return HC.IMAGE_APNG
+
+ else:
+
+ return HC.IMAGE_PNG
+
else:
@@ -253,7 +281,7 @@ def GetMime( path ):
try:
- mime = HydrusVideoHandling.GetMimeFromFFMPEG( path )
+ mime = HydrusVideoHandling.GetMime( path )
if mime != HC.APPLICATION_UNKNOWN:
diff --git a/include/HydrusNetworking.py b/include/HydrusNetworking.py
index 048d7758..7123d39e 100644
--- a/include/HydrusNetworking.py
+++ b/include/HydrusNetworking.py
@@ -105,14 +105,21 @@ class BandwidthRules( HydrusSerialisable.SerialisableBase ):
- def CanContinue( self, bandwidth_tracker, threshold = 60 ):
+ def CanContinue( self, bandwidth_tracker, threshold = 15 ):
with self._lock:
for ( bandwidth_type, time_delta, max_allowed ) in self._rules:
+ # Do not stop ongoing just because starts are throttled
+ requests_rule = bandwidth_type == HC.BANDWIDTH_TYPE_REQUESTS
+
# Do not block an ongoing jpg download because the current month is 100.03% used
- if time_delta is None or time_delta > threshold:
+ wait_is_too_long = time_delta is None or time_delta > threshold
+
+ ignore_rule = requests_rule or wait_is_too_long
+
+ if ignore_rule:
continue
@@ -127,19 +134,21 @@ class BandwidthRules( HydrusSerialisable.SerialisableBase ):
- def CanStart( self, bandwidth_tracker, threshold = 60 ):
+ def CanStart( self, bandwidth_tracker, threshold = 5 ):
with self._lock:
for ( bandwidth_type, time_delta, max_allowed ) in self._rules:
# Do not prohibit a new job from starting just because the current download speed is 210/200KB/s
- if time_delta is not None and time_delta < threshold:
+ ignore_rule_for_starting = bandwidth_type == HC.BANDWIDTH_TYPE_DATA and time_delta is not None and time_delta <= threshold
+
+ if ignore_rule_for_starting:
continue
- if bandwidth_tracker.GetUsage( bandwidth_type, time_delta ) > max_allowed:
+ if bandwidth_tracker.GetUsage( bandwidth_type, time_delta ) >= max_allowed:
return False
@@ -266,6 +275,11 @@ class BandwidthTracker( HydrusSerialisable.SerialisableBase ):
self._seconds_requests = counters[ 9 ]
+ def _GetCurrentDateTime( self ):
+
+ return datetime.datetime.utcfromtimestamp( HydrusData.GetNow() )
+
+
def _GetMonthTime( self, dt ):
( year, month ) = ( dt.year, dt.month )
@@ -281,7 +295,7 @@ class BandwidthTracker( HydrusSerialisable.SerialisableBase ):
if time_delta is None:
- dt = datetime.datetime.utcnow()
+ dt = self._GetCurrentDateTime()
month_time = self._GetMonthTime( dt )
@@ -473,7 +487,7 @@ class BandwidthTracker( HydrusSerialisable.SerialisableBase ):
with self._lock:
- dt = datetime.datetime.utcnow()
+ dt = self._GetCurrentDateTime()
( month_time, day_time, hour_time, minute_time, second_time ) = self._GetTimes( dt )
@@ -495,7 +509,7 @@ class BandwidthTracker( HydrusSerialisable.SerialisableBase ):
with self._lock:
- dt = datetime.datetime.utcnow()
+ dt = self._GetCurrentDateTime()
( month_time, day_time, hour_time, minute_time, second_time ) = self._GetTimes( dt )
diff --git a/include/HydrusSerialisable.py b/include/HydrusSerialisable.py
index 058c8184..50dbae96 100644
--- a/include/HydrusSerialisable.py
+++ b/include/HydrusSerialisable.py
@@ -48,6 +48,9 @@ SERIALISABLE_TYPE_APPLICATION_COMMAND = 42
SERIALISABLE_TYPE_DUPLICATE_ACTION_OPTIONS = 43
SERIALISABLE_TYPE_TAG_CENSOR = 44
SERIALISABLE_TYPE_BANDWIDTH_MANAGER = 45
+SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER = 46
+SERIALISABLE_TYPE_NETWORK_CONTEXT = 47
+SERIALISABLE_TYPE_NETWORK_LOGIN_MANAGER = 48
SERIALISABLE_TYPES_TO_OBJECT_TYPES = {}
diff --git a/include/HydrusVideoHandling.py b/include/HydrusVideoHandling.py
index 68fc1422..6cababfb 100755
--- a/include/HydrusVideoHandling.py
+++ b/include/HydrusVideoHandling.py
@@ -27,6 +27,18 @@ if not os.path.exists( FFMPEG_PATH ):
FFMPEG_PATH = os.path.basename( FFMPEG_PATH )
+def CheckFFMPEGError( lines ):
+
+ if "No such file or directory" in lines[-1]:
+
+ raise IOError( "File not found!" )
+
+
+ if 'Invalid data' in lines[-1]:
+
+ raise HydrusExceptions.MimeException( 'FFMPEG could not parse.' )
+
+
def GetFFMPEGVersion():
# open the file in a pipe, provoke an error, read output
@@ -78,19 +90,158 @@ def GetFFMPEGVersion():
return 'unknown'
+# bits of this were originally cribbed from moviepy
+def GetFFMPEGInfoLines( path, count_frames_manually = False ):
+
+ # open the file in a pipe, provoke an error, read output
+
+ try:
+
+ path.encode( 'ascii' ) # throwing unicode at the console is a mess best left for Python 3
+
+ except UnicodeEncodeError:
+
+ ( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
+
+ with open( path, 'rb' ) as source:
+
+ with open( temp_path, 'wb' ) as dest:
+
+ HydrusPaths.CopyFileLikeToFileLike( source, dest )
+
+
+
+ try:
+
+ return GetFFMPEGInfoLines( temp_path, count_frames_manually )
+
+ finally:
+
+ HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
+
+
+
+ cmd = [ FFMPEG_PATH, "-i", path ]
+
+ if count_frames_manually:
+
+ if HC.PLATFORM_WINDOWS:
+
+ cmd += [ "-f", "null", "NUL" ]
+
+ else:
+
+ cmd += [ "-f", "null", "/dev/null" ]
+
+
+
+ try:
+
+ proc = subprocess.Popen( cmd, bufsize = 10**5, stdout = subprocess.PIPE, stderr = subprocess.PIPE, startupinfo = HydrusData.GetHideTerminalSubprocessStartupInfo() )
+
+ except:
+
+ if not os.path.exists( FFMPEG_PATH ):
+
+ raise Exception( 'FFMPEG was not found!' )
+
+ else:
+
+ raise
+
+
+
+ raw_info = proc.stderr.read()
+
+ try:
+
+ info = raw_info.decode( 'utf8' )
+
+ except UnicodeDecodeError:
+
+ info = raw_info
+
+
+ proc.wait()
+
+ proc.communicate()
+
+ del proc
+
+ lines = info.splitlines()
+
+ CheckFFMPEGError( lines )
+
+ return lines
+
def GetFFMPEGVideoProperties( path, count_frames_manually = False ):
- info = Hydrusffmpeg_parse_infos( path, count_frames_manually = count_frames_manually )
+ lines = GetFFMPEGInfoLines( path )
- ( w, h ) = info[ 'video_size' ]
+ if not ParseFFMPEGHasVideo( lines ):
+
+ raise HydrusExceptions.MimeException( 'File did not appear to have a video stream!' )
+
- duration_in_s = info[ 'duration' ]
+ resolution = ParseFFMPEGVideoResolution( lines )
- duration = int( duration_in_s * 1000 )
+ duration = ParseFFMPEGDuration( lines )
- num_frames = info[ 'video_nframes' ]
+ if duration is None:
+
+ fps = ParseFFMPEGFPS( lines )
+
+ if fps is None:
+
+ raise HydrusExceptions.MimeException( 'Could not determine either the duration or fps!' )
+
+
+ if not count_frames_manually:
+
+ count_frames_manually = True
+
+ lines = GetFFMPEGInfoLines( path, count_frames_manually )
+
+
+ num_frames = ParseFFMPEGNumFramesManually( lines )
+
+ duration = num_frames / fps
+
+ else:
+
+ if not count_frames_manually:
+
+ fps = ParseFFMPEGFPS( lines )
+
+ it_was_accurate = fps is not None
+
+ if it_was_accurate:
+
+ num_frames = duration * fps
+
+ if num_frames != int( num_frames ): # we want whole numbers--anything else suggests start_offset is off or whatever
+
+ it_was_accurate = False
+
+
+
+ if not it_was_accurate:
+
+ count_frames_manually = True
+
+ lines = GetFFMPEGInfoLines( path, count_frames_manually )
+
+
+
+ if count_frames_manually:
+
+ num_frames = ParseFFMPEGNumFramesManually( lines )
+
+
- return ( ( w, h ), duration, num_frames )
+ duration_in_ms = int( duration * 1000 )
+
+ return ( resolution, duration_in_ms, num_frames )
def GetMatroskaOrWebm( path ):
@@ -143,46 +294,50 @@ def GetMatroskaOrWebMProperties( path ):
return ( ( width, height ), duration, num_frames )
-def GetMimeFromFFMPEG( path ):
+def GetMime( path ):
- info = Hydrusffmpeg_parse_infos( path )
+ lines = GetFFMPEGInfoLines( path )
- if 'mime_text' in info:
+ try:
- mime_text = info[ 'mime_text' ]
+ mime_text = ParseFFMPEGMimeText( lines )
- if 'matroska' in mime_text or 'webm' in mime_text:
+ except HydrusExceptions.MimeException:
+
+ return HC.APPLICATION_UNKNOWN
+
+
+ if 'matroska' in mime_text or 'webm' in mime_text:
+
+ # typically it is 'matroska,webm'
+
+ return GetMatroskaOrWebm( path )
+
+ elif mime_text in ( 'mpeg', 'mpegvideo', 'mpegts' ):
+
+ return HC.VIDEO_MPEG
+
+ elif mime_text == 'flac':
+
+ return HC.AUDIO_FLAC
+
+ elif mime_text == 'mp3':
+
+ return HC.AUDIO_MP3
+
+ elif mime_text == 'ogg':
+
+ return HC.AUDIO_OGG
+
+ elif mime_text == 'asf':
+
+ if ParseFFMPEGHasVideo( lines ):
- # typically it is 'matroska,webm'
+ return HC.VIDEO_WMV
- return GetMatroskaOrWebm( path )
+ else:
- elif mime_text in ( 'mpeg', 'mpegvideo', 'mpegts' ):
-
- return HC.VIDEO_MPEG
-
- elif mime_text == 'flac':
-
- return HC.AUDIO_FLAC
-
- elif mime_text == 'mp3':
-
- return HC.AUDIO_MP3
-
- elif mime_text == 'ogg':
-
- return HC.AUDIO_OGG
-
- elif mime_text == 'asf':
-
- if info[ 'video_found' ]:
-
- return HC.VIDEO_WMV
-
- else:
-
- return HC.AUDIO_WMA
-
+ return HC.AUDIO_WMA
@@ -190,122 +345,38 @@ def GetMimeFromFFMPEG( path ):
def HasVideoStream( path ):
- try:
-
- info = Hydrusffmpeg_parse_infos( path )
-
- except IOError as e:
-
- HydrusData.ShowException( 'Determining the mime for the file at ' + path + ' caused the following problem:' )
- HydrusData.ShowException( e )
-
- return False
-
+ lines = GetFFMPEGInfoLines( path )
- return info[ 'video_found' ]
+ return ParseFFMPEGHasVideo( lines )
-# this is cribbed from moviepy
-def Hydrusffmpeg_parse_infos(filename, print_infos=False, count_frames_manually = False ):
- """Get file infos using ffmpeg.
-
- Returns a dictionnary with the fields:
- "video_found", "video_fps", "duration", "video_nframes",
- "video_duration"
- "audio_found", "audio_fps"
-
- "video_duration" is slightly smaller than "duration" to avoid
- fetching the uncomplete frames at the end, which raises an error.
-
- """
+def ParseFFMPEGAudio( lines ):
- # open the file in a pipe, provoke an error, read output
+ # this is from the old stuff--might be helpful later when we add audio
- try:
-
- filename.encode( 'ascii' ) # throwing unicode at the console is a mess best left for Python 3
-
- except UnicodeEncodeError:
-
- ( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
-
- with open( filename, 'rb' ) as source:
-
- with open( temp_path, 'wb' ) as dest:
-
- HydrusPaths.CopyFileLikeToFileLike( source, dest )
-
-
-
+ lines_audio = [l for l in lines if ' Audio: ' in l]
+
+ audio_found = lines_audio != []
+
+ if audio_found:
+ line = lines_audio[0]
try:
-
- return Hydrusffmpeg_parse_infos( temp_path )
-
- finally:
-
- HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
-
-
+ match = re.search(" [0-9]* Hz", line)
+ audio_fps = int(line[match.start()+1:match.end()])
+ except:
+ audio_fps = 'unknown'
- cmd = [ FFMPEG_PATH, "-i", filename ]
-
- is_GIF = filename.endswith('.gif')
-
- doing_manual_frame_count = is_GIF or count_frames_manually
-
- if doing_manual_frame_count:
- if HC.PLATFORM_WINDOWS: cmd += ["-f", "null", "NUL"]
- else: cmd += ["-f", "null", "/dev/null"]
-
- try:
-
- proc = subprocess.Popen( cmd, bufsize=10**5, stdout=subprocess.PIPE, stderr=subprocess.PIPE, startupinfo = HydrusData.GetHideTerminalSubprocessStartupInfo() )
-
- except:
-
- if not os.path.exists( FFMPEG_PATH ):
-
- raise Exception( 'FFMPEG was not found!' )
-
- else:
-
- raise
-
-
-
- raw_infos = proc.stderr.read()
-
- try:
-
- infos = raw_infos.decode( 'utf8' )
-
- except UnicodeDecodeError:
-
- infos = raw_infos
-
-
- proc.wait()
-
- proc.communicate()
-
- del proc
-
- if print_infos:
- # print the whole info text returned by FFMPEG
- HydrusData.Print( infos )
-
- lines = infos.splitlines()
- if "No such file or directory" in lines[-1]:
- raise IOError("%s not found ! Wrong path ?"%filename)
- if 'Invalid data' in lines[-1]:
- raise HydrusExceptions.MimeException( 'FFMPEG could not parse.' )
-
- result = dict()
+def ParseFFMPEGDuration( lines ):
# get duration (in seconds)
# Duration: 00:00:02.46, start: 0.033000, bitrate: 1069 kb/s
try:
- keyword = ('frame=' if is_GIF else 'Duration: ')
- line = [l for l in lines if keyword in l][0]
+
+ line = [ l for l in lines if 'Duration:' in l ][0]
+
+ if 'Duration: N/A' in line:
+
+ return None
+
if 'start:' in line:
@@ -326,18 +397,85 @@ def Hydrusffmpeg_parse_infos(filename, print_infos=False, count_frames_manually
match = re.search("[0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9][0-9]", line)
hms = map(float, line[match.start()+1:match.end()].split(':'))
- if len(hms) == 1:
- result['duration'] = hms[0]
- elif len(hms) == 2:
- result['duration'] = 60*hms[0]+hms[1]
- elif len(hms) ==3:
- result['duration'] = 3600*hms[0]+60*hms[1]+hms[2]
+ if len( hms ) == 1:
+
+ duration = hms[0]
+
+ elif len( hms ) == 2:
+
+ duration = 60 * hms[0] + hms[1]
+
+ elif len( hms ) ==3:
+
+ duration = 3600 * hms[0] + 60 * hms[1] + hms[2]
+
- result[ 'duration' ] -= start_offset
+ duration -= start_offset
+
+ return duration
except:
- raise IOError("Error reading duration in file %s,"%(filename)+
- "Text parsed: %s"%infos)
+
+ raise HydrusExceptions.MimeException( 'Error reading duration!' )
+
+
+def ParseFFMPEGFPS( lines ):
+
+ try:
+
+ line = ParseFFMPEGVideoLine( lines )
+
+ # get the frame rate
+
+ match = re.search("( [0-9]*.| )[0-9]* tbr", line)
+
+ if match is not None:
+
+ fps = line[match.start():match.end()].split(' ')[1]
+
+
+ tbr_fps_is_likely_garbage = match is None or fps.endswith( 'k' ) or float( fps ) > 60
+
+ if tbr_fps_is_likely_garbage:
+
+ match = re.search("( [0-9]*.| )[0-9]* fps", line)
+
+ if match is not None:
+
+ fps = line[match.start():match.end()].split(' ')[1]
+
+
+ fps_is_likely_garbage = match is None or fps.endswith( 'k' ) or float( fps ) > 60
+
+ if fps_is_likely_garbage:
+
+ return None
+
+
+
+ fps = float( fps )
+
+ return fps
+
+ except:
+
+ raise HydrusExceptions.MimeException( 'Error estimating framerate!' )
+
+
+def ParseFFMPEGHasVideo( lines ):
+
+ try:
+
+ video_line = ParseFFMPEGVideoLine( lines )
+
+ except HydrusExceptions.MimeException:
+
+ return False
+
+
+ return True
+
+def ParseFFMPEGMimeText( lines ):
try:
@@ -349,116 +487,66 @@ def Hydrusffmpeg_parse_infos(filename, print_infos=False, count_frames_manually
mime_text = text.split( ', from' )[0]
- result[ 'mime_text' ] = mime_text
+ return mime_text
except:
- pass
+ raise HydrusExceptions.MimeException( 'Error reading mime!' )
- if count_frames_manually:
+def ParseFFMPEGNumFramesManually( lines ):
+
+ try:
frame_lines = [ l for l in lines if l.startswith( 'frame= ' ) ]
- if len( frame_lines ) > 0:
+ l = frame_lines[-1] # there will be several of these, counting up as the file renders. we hence want the final one
+
+ while ' ' in l:
- l = frame_lines[-1] # there will be several of these, counting up as the file renders. we hence want the final one
-
- while ' ' in l:
-
- l = l.replace( ' ', ' ' )
-
-
- num_frames = int( l.split( ' ' )[1] )
-
- result[ 'video_nframes' ] = num_frames
+ l = l.replace( ' ', ' ' )
+ num_frames = int( l.split( ' ' )[1] )
+
+ return num_frames
+
+ except:
+
+ raise HydrusExceptions.MimeException( 'Error counting number of frames!' )
+
+
+def ParseFFMPEGVideoLine( lines ):
# get the output line that speaks about video
lines_video = [ l for l in lines if ' Video: ' in l and not ( ' Video: png' in l or ' Video: jpg' in l ) ] # mp3 says it has a 'png' video stream
- result['video_found'] = ( lines_video != [] )
-
- if result['video_found']:
+ if len( lines_video ) == 0:
+
+ raise HydrusExceptions.MimeException( 'Could not find video information!' )
+
+
+ line = lines_video[0]
+
+ return line
+
+def ParseFFMPEGVideoResolution( lines ):
+
+ try:
+
+ line = ParseFFMPEGVideoLine( lines )
- line = lines_video[0]
-
# get the size, of the form 460x320 (w x h)
match = re.search(" [0-9]*x[0-9]*(,| )", line)
- s = list(map(int, line[match.start():match.end()-1].split('x')))
- result['video_size'] = s
- have_to_fetch_manually = False
+ resolution = list(map(int, line[match.start():match.end()-1].split('x')))
- if 'video_nframes' in result:
-
- result[ 'video_fps' ] = result[ 'video_nframes' ] / result[ 'duration' ]
-
- else:
-
- # get the frame rate
-
- match = re.search("( [0-9]*.| )[0-9]* tbr", line)
-
- if match is not None:
-
- fps = line[match.start():match.end()].split(' ')[1]
-
-
- if match is None or fps.endswith( 'k' ):
-
- match = re.search("( [0-9]*.| )[0-9]* fps", line)
-
- if match is not None:
-
- fps = line[match.start():match.end()].split(' ')[1]
-
-
- if match is None or fps.endswith( 'k' ) or float( fps ) > 60:
-
- if not doing_manual_frame_count:
-
- return Hydrusffmpeg_parse_infos( filename, count_frames_manually = True )
-
- else:
-
- raise Exception( 'Could not determine framerate!' )
-
-
-
-
- result['video_fps'] = float( fps )
-
- num_frames = result['duration'] * result['video_fps']
-
- if num_frames != int( num_frames ):
-
- return Hydrusffmpeg_parse_infos( filename, count_frames_manually = True )
-
-
- result['video_nframes'] = int( num_frames )
-
+ return resolution
+
+ except:
+
+ raise HydrusExceptions.MimeException( 'Error counting number of frames!' )
- result['video_duration'] = result['duration']
- # We could have also recomputed the duration from the number
- # of frames, as follows:
- # >>> result['video_duration'] = result['video_nframes'] / result['video_fps']
-
-
- lines_audio = [l for l in lines if ' Audio: ' in l]
-
- result['audio_found'] = lines_audio != []
-
- if result['audio_found']:
- line = lines_audio[0]
- try:
- match = re.search(" [0-9]* Hz", line)
- result['audio_fps'] = int(line[match.start()+1:match.end()])
- except:
- result['audio_fps'] = 'unknown'
-
- return result
# This was built from moviepy's FFMPEG_VideoReader
class VideoRendererFFMPEG( object ):
@@ -518,7 +606,7 @@ class VideoRendererFFMPEG( object ):
self.close()
- if self._mime == HC.IMAGE_GIF:
+ if self._mime in ( HC.IMAGE_APNG, HC.IMAGE_GIF ):
ss = 0
self.pos = 0
diff --git a/include/TestClientDownloading.py b/include/TestClientDownloading.py
index 7429f876..d62a51ba 100644
--- a/include/TestClientDownloading.py
+++ b/include/TestClientDownloading.py
@@ -91,7 +91,7 @@ class TestDownloaders( unittest.TestCase ):
with open( os.path.join( HC.STATIC_DIR, 'testing', 'pixiv_gallery.html' ) ) as f: pixiv_gallery = f.read()
with open( os.path.join( HC.STATIC_DIR, 'testing', 'pixiv_image_page.html' ) ) as f: pixiv_page = f.read()
- HG.test_controller.GetHTTP().SetResponse( HC.GET, 'http://www.pixiv.net/search.php?word=naruto&s_mode=s_tag_full&order=date_d&p=1', pixiv_gallery )
+ HG.test_controller.GetHTTP().SetResponse( HC.GET, 'https://www.pixiv.net/search.php?word=naruto&s_mode=s_tag_full&order=date_d&p=1', pixiv_gallery )
HG.test_controller.GetHTTP().SetResponse( HC.GET, 'http://www.pixiv.net/member_illust.php?mode=medium&illust_id=50926312', pixiv_page )
HG.test_controller.GetHTTP().SetResponse( HC.GET, 'http://i3.pixiv.net/img-original/img/2014/01/25/19/21/56/41171994_p0.jpg', 'image file' )
@@ -106,7 +106,7 @@ class TestDownloaders( unittest.TestCase ):
# a manga one, http://www.pixiv.net/member_illust.php?mode=medium&illust_id=51078392, is currently filtered
- expected_gallery_urls = [ u'http://www.pixiv.net/member_illust.php?mode=medium&illust_id=50926312', 'a bunch of others' ]
+ expected_gallery_urls = [ u'https://www.pixiv.net/member_illust.php?mode=medium&illust_id=50926312', 'a bunch of others' ]
self.assertEqual( page_of_urls[0], expected_gallery_urls[0] )
diff --git a/include/TestClientNetworking.py b/include/TestClientNetworking.py
new file mode 100644
index 00000000..6e8bee81
--- /dev/null
+++ b/include/TestClientNetworking.py
@@ -0,0 +1,454 @@
+import ClientConstants as CC
+import ClientNetworking
+import collections
+import HydrusConstants as HC
+import HydrusData
+import HydrusNetworking
+import os
+import TestConstants
+import threading
+import time
+import unittest
+import HydrusGlobals as HG
+from httmock import all_requests, urlmatch, HTTMock
+from mock import patch
+
+# some gumpf
+GOOD_RESPONSE = ''.join( chr( i ) for i in range( 256 ) )
+
+# 256KB of gumpf
+LONG_GOOD_RESPONSE = GOOD_RESPONSE * 4 * 256
+
+@all_requests
+def catch_all( url, request ):
+
+ raise Exception( 'An unexpected request for ' + url + ' came through in testing.' )
+
+
+MOCK_DOMAIN = 'wew.lad'
+MOCK_SUBDOMAIN = 'top.wew.lad'
+MOCK_URL = 'https://wew.lad/folder/request&key1=value1&key2=value2'
+MOCK_SUBURL = 'https://top.wew.lad/folder2/request&key1=value1&key2=value2'
+
+@urlmatch( netloc = 'wew.lad' )
+def catch_wew_ok( url, request ):
+
+ return GOOD_RESPONSE
+
+@urlmatch( netloc = '123.45.67.89:45871' )
+def catch_hydrus_ok( url, request ):
+
+ return GOOD_RESPONSE
+
+class TestBandwidthManager( unittest.TestCase ):
+
+ def test_can_start( self ):
+
+ EMPTY_RULES = HydrusNetworking.BandwidthRules()
+
+ PERMISSIVE_DATA_RULES = HydrusNetworking.BandwidthRules()
+
+ PERMISSIVE_DATA_RULES.AddRule( HC.BANDWIDTH_TYPE_DATA, None, 1048576 )
+
+ PERMISSIVE_REQUEST_RULES = HydrusNetworking.BandwidthRules()
+
+ PERMISSIVE_REQUEST_RULES.AddRule( HC.BANDWIDTH_TYPE_REQUESTS, None, 10000 )
+
+ RESTRICTIVE_DATA_RULES = HydrusNetworking.BandwidthRules()
+
+ RESTRICTIVE_DATA_RULES.AddRule( HC.BANDWIDTH_TYPE_DATA, None, 10 )
+
+ RESTRICTIVE_REQUEST_RULES = HydrusNetworking.BandwidthRules()
+
+ RESTRICTIVE_REQUEST_RULES.AddRule( HC.BANDWIDTH_TYPE_REQUESTS, None, 1 )
+
+ DOMAIN_NETWORK_CONTEXT = ClientNetworking.NetworkContext( CC.NETWORK_CONTEXT_DOMAIN, MOCK_DOMAIN )
+ SUBDOMAIN_NETWORK_CONTEXT = ClientNetworking.NetworkContext( CC.NETWORK_CONTEXT_DOMAIN, MOCK_SUBDOMAIN )
+
+ GLOBAL_NETWORK_CONTEXTS = [ ClientNetworking.GLOBAL_NETWORK_CONTEXT ]
+ DOMAIN_NETWORK_CONTEXTS = [ ClientNetworking.GLOBAL_NETWORK_CONTEXT, DOMAIN_NETWORK_CONTEXT ]
+ SUBDOMAIN_NETWORK_CONTEXTS = [ ClientNetworking.GLOBAL_NETWORK_CONTEXT, DOMAIN_NETWORK_CONTEXT, SUBDOMAIN_NETWORK_CONTEXT ]
+ #
+
+ fast_forward = HydrusData.GetNow() + 3600
+
+ with patch.object( HydrusData, 'GetNow', return_value = fast_forward ):
+
+ bm = ClientNetworking.NetworkBandwidthManager()
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ #
+
+ bm.ReportRequestUsed( DOMAIN_NETWORK_CONTEXTS )
+ bm.ReportDataUsed( DOMAIN_NETWORK_CONTEXTS, 50 )
+ bm.ReportRequestUsed( SUBDOMAIN_NETWORK_CONTEXTS )
+ bm.ReportDataUsed( SUBDOMAIN_NETWORK_CONTEXTS, 25 )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ #
+
+ bm.SetRules( None, EMPTY_RULES )
+ bm.SetRules( MOCK_DOMAIN, EMPTY_RULES )
+ bm.SetRules( MOCK_SUBDOMAIN, EMPTY_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ bm.SetRules( None, PERMISSIVE_DATA_RULES )
+ bm.SetRules( MOCK_DOMAIN, PERMISSIVE_DATA_RULES )
+ bm.SetRules( MOCK_SUBDOMAIN, PERMISSIVE_DATA_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ bm.SetRules( None, PERMISSIVE_REQUEST_RULES )
+ bm.SetRules( MOCK_DOMAIN, PERMISSIVE_REQUEST_RULES )
+ bm.SetRules( MOCK_SUBDOMAIN, PERMISSIVE_REQUEST_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ #
+
+ bm.SetRules( MOCK_SUBDOMAIN, RESTRICTIVE_DATA_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ bm.SetRules( MOCK_SUBDOMAIN, RESTRICTIVE_REQUEST_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ bm.SetRules( MOCK_SUBDOMAIN, PERMISSIVE_REQUEST_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ #
+
+ bm.SetRules( MOCK_DOMAIN, RESTRICTIVE_DATA_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ bm.SetRules( MOCK_DOMAIN, RESTRICTIVE_REQUEST_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ bm.SetRules( MOCK_DOMAIN, PERMISSIVE_REQUEST_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( MOCK_SUBDOMAIN ) )
+
+ #
+
+ bm.SetRules( None, RESTRICTIVE_DATA_RULES )
+
+ self.assertFalse( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ bm.SetRules( None, RESTRICTIVE_REQUEST_RULES )
+
+ self.assertFalse( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+ bm.SetRules( None, PERMISSIVE_REQUEST_RULES )
+
+ self.assertTrue( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertTrue( bm.CanStart( MOCK_SUBDOMAIN ) )
+
+ # add some rules for all
+
+ bm.SetRules( None, RESTRICTIVE_DATA_RULES )
+ bm.SetRules( MOCK_DOMAIN, RESTRICTIVE_REQUEST_RULES )
+ bm.SetRules( MOCK_DOMAIN, EMPTY_RULES )
+
+ self.assertFalse( bm.CanStart( GLOBAL_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( DOMAIN_NETWORK_CONTEXTS ) )
+ self.assertFalse( bm.CanStart( SUBDOMAIN_NETWORK_CONTEXTS ) )
+
+
+
+ def test_can_continue( self ):
+
+ pass
+
+
+class TestNetworkingEngine( unittest.TestCase ):
+
+ def test_engine_shutdown_app( self ):
+
+ mock_controller = TestConstants.MockController()
+ bandwidth_manager = ClientNetworking.NetworkBandwidthManager()
+ session_manager = ClientNetworking.NetworkSessionManager()
+ login_manager = ClientNetworking.NetworkLoginManager()
+
+ engine = ClientNetworking.NetworkEngine( mock_controller, bandwidth_manager, session_manager, login_manager )
+
+ self.assertFalse( engine.IsRunning() )
+ self.assertFalse( engine.IsShutdown() )
+
+ engine.Start()
+
+ time.sleep( 0.1 )
+
+ self.assertTrue( engine.IsRunning() )
+ self.assertFalse( engine.IsShutdown() )
+
+ mock_controller.model_is_shutdown = True
+
+ engine._new_work_to_do.set()
+
+ time.sleep( 0.1 )
+
+ self.assertFalse( engine.IsRunning() )
+ self.assertTrue( engine.IsShutdown() )
+
+
+ def test_engine_shutdown_manual( self ):
+
+ mock_controller = TestConstants.MockController()
+ bandwidth_manager = ClientNetworking.NetworkBandwidthManager()
+ session_manager = ClientNetworking.NetworkSessionManager()
+ login_manager = ClientNetworking.NetworkLoginManager()
+
+ engine = ClientNetworking.NetworkEngine( mock_controller, bandwidth_manager, session_manager, login_manager )
+
+ self.assertFalse( engine.IsRunning() )
+ self.assertFalse( engine.IsShutdown() )
+
+ engine.Start()
+
+ time.sleep( 0.1 )
+
+ self.assertTrue( engine.IsRunning() )
+ self.assertFalse( engine.IsShutdown() )
+
+ engine.Shutdown()
+
+ time.sleep( 0.1 )
+
+ self.assertFalse( engine.IsRunning() )
+ self.assertTrue( engine.IsShutdown() )
+
+
+class TestNetworkingJob( unittest.TestCase ):
+
+ def _GetJob( self ):
+
+ job = ClientNetworking.NetworkJob( 'GET', MOCK_URL )
+
+ mock_controller = TestConstants.MockController()
+ bandwidth_manager = ClientNetworking.NetworkBandwidthManager()
+ session_manager = ClientNetworking.NetworkSessionManager()
+ login_manager = ClientNetworking.NetworkLoginManager()
+
+ engine = ClientNetworking.NetworkEngine( mock_controller, bandwidth_manager, session_manager, login_manager )
+
+ job.engine = engine
+
+ return job
+
+
+ def test_cancelled_manually( self ):
+
+ job = self._GetJob()
+
+ self.assertFalse( job.IsCancelled() )
+ self.assertFalse( job.IsDone() )
+
+ job.Cancel()
+
+ self.assertTrue( job.IsCancelled() )
+ self.assertTrue( job.IsDone() )
+
+
+ def test_cancelled_app_shutdown( self ):
+
+ job = self._GetJob()
+
+ self.assertFalse( job.IsCancelled() )
+ self.assertFalse( job.IsDone() )
+
+ job.engine.controller.model_is_shutdown = True
+
+ self.assertTrue( job.IsCancelled() )
+ self.assertTrue( job.IsDone() )
+
+
+ def test_sleep( self ):
+
+ job = self._GetJob()
+
+ self.assertFalse( job.IsAsleep() )
+
+ job.Sleep( 3 )
+
+ self.assertTrue( job.IsAsleep() )
+
+ five_secs_from_now = HydrusData.GetNow() + 5
+
+ with patch.object( HydrusData, 'GetNow', return_value = five_secs_from_now ):
+
+ self.assertFalse( job.IsAsleep() )
+
+
+
+class TestNetworkingJobWeb( unittest.TestCase ):
+
+ def _GetJob( self ):
+
+ job = ClientNetworking.NetworkJob( 'GET', MOCK_URL )
+
+ controller = TestConstants.MockController()
+
+ job.controller = controller
+
+ return job
+
+
+ def test_bandwidth_ok( self ):
+
+ # test bandwidth override
+
+ # test bandwidth ok
+ # test it not ok
+
+ # repeat for the login one
+
+ pass
+
+
+ def test_done_ok( self ):
+
+ return # need to flush out session, bandwidth, login code
+
+ with HTTMock( catch_all ):
+
+ with HTTMock( catch_wew_ok ):
+
+ job = self._GetJob()
+
+ job.Start()
+
+ self.assertFalse( job.HasError() )
+
+ self.assertEqual( job.GetContent(), GOOD_RESPONSE )
+
+
+
+
+ def test_error( self ):
+
+ job = self._GetJob()
+
+ # do a requests job that cancels
+
+ # haserror
+ # geterrorexception
+ # geterrortext
+
+
+ def test_generate_login_process( self ):
+
+ # test the system works as expected
+
+ pass
+
+
+ def test_needs_login( self ):
+
+ # test for both normal and login
+
+ pass
+
+
+class TestNetworkingJobHydrus( unittest.TestCase ):
+
+ def _GetJob( self ):
+
+ job = ClientNetworking.NetworkJob( 'GET', 'https://123.45.67.89:45871/muh_hydrus_command' )
+
+ controller = TestConstants.MockController()
+
+ job.controller = controller
+
+ return job
+
+
+ def test_bandwidth_ok( self ):
+
+ # test bandwidth override
+
+ # test bandwidth ok
+ # test it not ok
+
+ # repeat for the login one
+
+ pass
+
+
+ def test_done_ok( self ):
+
+ return # need to flush out session, bandwidth, login code
+
+ with HTTMock( catch_all ):
+
+ with HTTMock( catch_hydrus_ok ):
+
+ job = self._GetJob()
+
+ job.Start()
+
+ self.assertFalse( job.HasError() )
+
+ self.assertEqual( job.GetContent(), GOOD_RESPONSE )
+
+
+
+
+ def test_error( self ):
+
+ job = self._GetJob()
+
+ # do a requests job that cancels
+
+ # haserror
+ # geterrorexception
+ # geterrortext
+
+
+ def test_generate_login_process( self ):
+
+ # test the system works as expected
+
+ pass
+
+
+ def test_needs_login( self ):
+
+ # test for both normal and login
+
+ pass
+
+
diff --git a/include/TestConstants.py b/include/TestConstants.py
index aea9d0d5..2ee35a82 100644
--- a/include/TestConstants.py
+++ b/include/TestConstants.py
@@ -1,12 +1,14 @@
import collections
import ClientConstants as CC
import HydrusConstants as HC
+import HydrusGlobals as HG
import HydrusTags
import os
import random
import threading
import weakref
import HydrusData
+import HydrusThreading
import wx
DB_DIR = None
@@ -27,6 +29,23 @@ def ConvertServiceKeysToContentUpdatesToComparable( service_keys_to_content_upda
return comparable_dict
+class MockController( object ):
+
+ def __init__( self ):
+
+ self.model_is_shutdown = False
+
+
+ def CallToThread( self, callable, *args, **kwargs ):
+
+ return HG.test_controller.CallToThread( callable, *args, **kwargs )
+
+
+ def ModelIsShutdown( self ):
+
+ return self.model_is_shutdown
+
+
class FakeHTTPConnectionManager():
def __init__( self ):
diff --git a/include/TestDB.py b/include/TestDB.py
index 90505d43..86de3b7a 100644
--- a/include/TestDB.py
+++ b/include/TestDB.py
@@ -685,6 +685,7 @@ class TestClientDB( unittest.TestCase ):
test_files.append( ( 'muh_webm.webm', '55b6ce9d067326bf4b2fbe66b8f51f366bc6e5f776ba691b0351364383c43fcb', 84069, HC.VIDEO_WEBM, 640, 360, 4010, 120, None ) )
test_files.append( ( 'muh_jpg.jpg', '5d884d84813beeebd59a35e474fa3e4742d0f2b6679faa7609b245ddbbd05444', 42296, HC.IMAGE_JPEG, 392, 498, None, None, None ) )
test_files.append( ( 'muh_png.png', 'cdc67d3b377e6e1397ffa55edc5b50f6bdf4482c7a6102c6f27fa351429d6f49', 31452, HC.IMAGE_PNG, 191, 196, None, None, None ) )
+ test_files.append( ( 'muh_apng.png', '9e7b8b5abc7cb11da32db05671ce926a2a2b701415d1b2cb77a28deea51010c3', 616956, HC.IMAGE_APNG, 500, 500, 1880, 47, None ) )
test_files.append( ( 'muh_gif.gif', '00dd9e9611ebc929bfc78fde99a0c92800bbb09b9d18e0946cea94c099b211c2', 15660, HC.IMAGE_GIF, 329, 302, 600, 5, None ) )
for ( filename, hex_hash, size, mime, width, height, duration, num_frames, num_words ) in test_files:
diff --git a/include/TestHydrusNetworking.py b/include/TestHydrusNetworking.py
index bde2ffb6..689254c1 100644
--- a/include/TestHydrusNetworking.py
+++ b/include/TestHydrusNetworking.py
@@ -1,106 +1,524 @@
import collections
import HydrusConstants as HC
import os
+import random
import TestConstants
import time
import unittest
import HydrusData
import HydrusGlobals as HG
import HydrusNetworking
+from mock import patch
-class TestBandwidthManagement( unittest.TestCase ):
+now = HydrusData.GetNow()
+
+now_10 = now + 10
+
+now_20 = now + 20
+
+with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ HIGH_USAGE = HydrusNetworking.BandwidthTracker()
+
+ for i in range( 100 ):
+
+ HIGH_USAGE.ReportRequestUsed()
+ HIGH_USAGE.ReportDataUsed( random.randint( 512, 1024 ) )
+
+
+ LOW_USAGE = HydrusNetworking.BandwidthTracker()
+
+ LOW_USAGE.ReportRequestUsed()
+ LOW_USAGE.ReportDataUsed( 1024 )
+
+ ZERO_USAGE = HydrusNetworking.BandwidthTracker()
+
+class TestBandwidthRules( unittest.TestCase ):
+
+ def test_no_rules( self ):
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertTrue( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+
+ def test_per_sec( self ):
+
+ # at short time deltas, we can always start based on data alone
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ rules.AddRule( HC.BANDWIDTH_TYPE_DATA, 1, 10240 )
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertTrue( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertFalse( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_10 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertTrue( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_20 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertTrue( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ #
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ rules.AddRule( HC.BANDWIDTH_TYPE_REQUESTS, 1, 1 )
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertFalse( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_10 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertTrue( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_20 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertTrue( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ #
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ rules.AddRule( HC.BANDWIDTH_TYPE_DATA, 1, 10240 )
+ rules.AddRule( HC.BANDWIDTH_TYPE_REQUESTS, 1, 1 )
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertFalse( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertFalse( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_10 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertTrue( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_20 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertTrue( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+
+ def test_per_min( self ):
+
+ # cutoff is 15s for continue
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ rules.AddRule( HC.BANDWIDTH_TYPE_DATA, 60, 10240 )
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_10 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_20 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ #
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ rules.AddRule( HC.BANDWIDTH_TYPE_REQUESTS, 60, 10 )
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_10 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_20 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ #
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ rules.AddRule( HC.BANDWIDTH_TYPE_DATA, 60, 10240 )
+ rules.AddRule( HC.BANDWIDTH_TYPE_REQUESTS, 60, 10 )
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_10 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_20 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+
+ def test_per_month( self ):
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ rules.AddRule( HC.BANDWIDTH_TYPE_DATA, None, 10240 )
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_10 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_20 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ #
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ rules.AddRule( HC.BANDWIDTH_TYPE_REQUESTS, None, 10 )
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_10 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_20 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ #
+
+ rules = HydrusNetworking.BandwidthRules()
+
+ rules.AddRule( HC.BANDWIDTH_TYPE_DATA, None, 10240 )
+ rules.AddRule( HC.BANDWIDTH_TYPE_REQUESTS, None, 10 )
+
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_10 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+ with patch.object( HydrusData, 'GetNow', return_value = now_20 ):
+
+ self.assertTrue( rules.CanStart( ZERO_USAGE ) )
+ self.assertTrue( rules.CanStart( LOW_USAGE ) )
+ self.assertFalse( rules.CanStart( HIGH_USAGE ) )
+
+ self.assertTrue( rules.CanContinue( ZERO_USAGE ) )
+ self.assertTrue( rules.CanContinue( LOW_USAGE ) )
+ self.assertTrue( rules.CanContinue( HIGH_USAGE ) )
+
+
+
+class TestBandwidthTracker( unittest.TestCase ):
def test_bandwidth_tracker( self ):
bandwidth_tracker = HydrusNetworking.BandwidthTracker()
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 0 ), 0 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 0 ), 0 )
+ self.assertEqual( bandwidth_tracker.GetCurrentMonthSummary(), 'used 0.0B in 0 requests this month' )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 1 ), 0 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 1 ), 0 )
+ now = HydrusData.GetNow()
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 2 ), 0 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 2 ), 0 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 6 ), 0 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 6 ), 0 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 3600 ), 0 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 3600 ), 0 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 0 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 0 )
- self.assertAlmostEquals
- #
-
- bandwidth_tracker.ReportDataUsed( 1024 )
- bandwidth_tracker.ReportRequestUsed()
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 0 ), 0 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 0 ), 0 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 1 ), 170 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 1 ), 1 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 2 ), 85 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 2 ), 1 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 6 ), 1024 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 6 ), 1 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 3600 ), 1024 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 3600 ), 1 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 1024 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 1 )
+ with patch.object( HydrusData, 'GetNow', return_value = now ):
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 0 ), 0 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 0 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 1 ), 0 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 1 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 2 ), 0 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 2 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 6 ), 0 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 6 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 3600 ), 0 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 3600 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 0 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 0 )
+
+ #
+
+ bandwidth_tracker.ReportDataUsed( 1024 )
+ bandwidth_tracker.ReportRequestUsed()
+
+ self.assertEqual( bandwidth_tracker.GetCurrentMonthSummary(), 'used 1024B in 1 requests this month' )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 0 ), 0 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 0 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 1 ), 170 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 1 ), 1 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 2 ), 85 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 2 ), 1 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 6 ), 1024 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 6 ), 1 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 3600 ), 1024 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 3600 ), 1 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 1024 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 1 )
+
#
- time.sleep( 5 )
+ five_secs_from_now = now + 5
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 0 ), 0 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 0 ), 0 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 1 ), 42 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 1 ), 0 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 2 ), 85 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 2 ), 0 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 6 ), 1024 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 6 ), 1 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 3600 ), 1024 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 3600 ), 1 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 1024 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 1 )
-
- #
-
- bandwidth_tracker.ReportDataUsed( 32 )
- bandwidth_tracker.ReportRequestUsed()
-
- bandwidth_tracker.ReportDataUsed( 32 )
- bandwidth_tracker.ReportRequestUsed()
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 0 ), 0 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 0 ), 0 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 1 ), 53 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 1 ), 2 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 2 ), 90 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 2 ), 2 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 6 ), 1088 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 6 ), 3 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 3600 ), 1088 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 3600 ), 3 )
-
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 1088 )
- self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 3 )
+ with patch.object( HydrusData, 'GetNow', return_value = five_secs_from_now ):
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 0 ), 0 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 0 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 1 ), 42 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 1 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 2 ), 85 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 2 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 6 ), 1024 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 6 ), 1 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 3600 ), 1024 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 3600 ), 1 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 1024 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 1 )
+
+ #
+
+ bandwidth_tracker.ReportDataUsed( 32 )
+ bandwidth_tracker.ReportRequestUsed()
+
+ bandwidth_tracker.ReportDataUsed( 32 )
+ bandwidth_tracker.ReportRequestUsed()
+
+ self.assertEqual( bandwidth_tracker.GetCurrentMonthSummary(), 'used 1.1KB in 3 requests this month' )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 0 ), 0 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 0 ), 0 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 1 ), 53 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 1 ), 2 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 2 ), 90 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 2 ), 2 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 6 ), 1088 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 6 ), 3 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, 3600 ), 1088 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, 3600 ), 3 )
+
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_DATA, None ), 1088 )
+ self.assertEqual( bandwidth_tracker.GetUsage( HC.BANDWIDTH_TYPE_REQUESTS, None ), 3 )
+
diff --git a/include/TestHydrusServer.py b/include/TestHydrusServer.py
index 75c61f1c..1b601d6d 100644
--- a/include/TestHydrusServer.py
+++ b/include/TestHydrusServer.py
@@ -62,7 +62,7 @@ class TestServer( unittest.TestCase ):
cls._local_booru = ClientServices.GenerateService( HydrusData.GenerateKey(), HC.LOCAL_BOORU, 'local booru' )
- services_manager = HG.test_controller.GetServicesManager()
+ services_manager = HG.test_controller.services_manager
services_manager._keys_to_services[ cls._clientside_file_service.GetServiceKey() ] = cls._clientside_file_service
services_manager._keys_to_services[ cls._clientside_tag_service.GetServiceKey() ] = cls._clientside_tag_service
diff --git a/static/testing/muh_apng.png b/static/testing/muh_apng.png
new file mode 100644
index 00000000..a1424217
Binary files /dev/null and b/static/testing/muh_apng.png differ
diff --git a/test.py b/test.py
index fe36f89c..a71b3847 100644
--- a/test.py
+++ b/test.py
@@ -19,6 +19,7 @@ from include import TestClientConstants
from include import TestClientDaemons
from include import TestClientDownloading
from include import TestClientListBoxes
+from include import TestClientNetworking
from include import TestConstants
from include import TestDialogs
from include import TestDB
@@ -125,8 +126,8 @@ class Controller( object ):
self._managers = {}
- self._services_manager = ClientCaches.ServicesManager( self )
- self._client_files_manager = ClientCaches.ClientFilesManager( self )
+ self.services_manager = ClientCaches.ServicesManager( self )
+ self.client_files_manager = ClientCaches.ClientFilesManager( self )
self._client_session_manager = ClientCaches.HydrusSessionManager( self )
self._managers[ 'tag_censorship' ] = ClientCaches.TagCensorshipManager( self )
@@ -195,11 +196,6 @@ class Controller( object ):
def DoHTTP( self, *args, **kwargs ): return self._http.Request( *args, **kwargs )
- def GetClientFilesManager( self ):
-
- return self._client_files_manager
-
-
def GetClientSessionManager( self ):
return self._client_session_manager
@@ -222,11 +218,9 @@ class Controller( object ):
return HC.options
- def GetManager( self, manager_type ): return self._managers[ manager_type ]
-
- def GetServicesManager( self ):
+ def GetManager( self, manager_type ):
- return self._services_manager
+ return self._managers[ manager_type ]
def GetServerSessionManager( self ):
@@ -293,8 +287,9 @@ class Controller( object ):
suites.append( unittest.TestLoader().loadTestsFromModule( TestHydrusTags ) )
if run_all or only_run == 'db': suites.append( unittest.TestLoader().loadTestsFromModule( TestDB ) )
if run_all or only_run == 'downloading':
-
suites.append( unittest.TestLoader().loadTestsFromModule( TestClientDownloading ) )
+ if run_all or only_run == 'networking':
+ suites.append( unittest.TestLoader().loadTestsFromModule( TestClientNetworking ) )
suites.append( unittest.TestLoader().loadTestsFromModule( TestHydrusNetworking ) )
if run_all or only_run == 'gui':
suites.append( unittest.TestLoader().loadTestsFromModule( TestDialogs ) )
the hydrus database
+A hydrus client consists of three components:
+-
+
these components can be put on different drives
+Although an initial install will keep these parts together, it is possible to run the database on a fast drive but keep your files in cheap slow storage. And if you have a very large collection, you can even spread your files across multiple drives. It is not very technically difficult, but I do not recommend it for new users.
+pulling them apart
+As always, I recommend creating a backup before you try any of this, just in case it goes wrong.
+If you have multiple drives and would like to spread your database across them, please do not move the folders around yourself--the database has an internal 'knowledge' of where it thinks its thumbnail and file folders are, and if you move them while it is closed, it will throw 'missing path' errors as soon as it boots. The internal hydrus logic of relative and absolute paths is not always obvious, so it is easy to make mistakes, even if you think you know what you are doing. Instead, please do it through the gui:
+Go database->migration, giving you this dialog:
+
***Talk about what portable means.***
+These operations are simple, but they can take some time to complete.
+If you move your database, the program will have to shut down first. Before you boot up again, you will have to create a new program shortcut:
+informing the software that the database has moved
+A straight call to the client executable will look for a database in install_dir/db. If one is not found, it will create one. So, if you move your database and then try to run the client again, it will try to create a new empty database in the previous location!
+So, pass it a command line argument, like so:
+-
+
And it will instead use the given path. You can use any path that is valid in your system, but I would not advise using network locations and so on, as the database works best with some clever device locking calls these interfaces may not provide.
+Rather than typing the path out in a terminal every time you want to launch your external database, create a new shortcut with the argument in. Something like this:
+
Note that an install with an 'external' database no longer needs access to write to its own path, so you can store it anywhere you like. If you move it, just double-check your shortcuts are still good and you are done.
+finally
+Now you have a new database in one or more new locations, make sure to update your backup routine to follow these new paths!
+p.s. running multiple clients
+Since you now know how to tell the software about an external database, you can, if you like, run multiple clients from the same install (and if you previously had multiple install folders, now you can now just use the one). Just make multiple shortcuts to the same client executable but with different database directories. They can run at the same time. You'll save yourself a little memory and update-hassle.
+