diff --git a/docs/changelog.md b/docs/changelog.md
index 6146c1f6..c4e9131c 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -3,7 +3,36 @@
!!! note
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
-## [Version 488](https://github.com/hydrusnetwork/hydrus/releases/tag/v487)
+## [Version 489](https://github.com/hydrusnetwork/hydrus/releases/tag/v489)
+
+### downloader pages
+* greatly improved the status reporting for downloader pages. the way the little text updates on your file and gallery progress are generated and presented is overhauled, and tests are unified across the different downloader pages. you now get specific texts on all possible reasons the queue cannot currently process, such as the emergency pause states under the _network_ menu or specific info like hitting the file limit, and all the code involved here is much cleaner
+* the 'working/pending' status, when you have a whole bunch of galleries or watchers wanting to run at the same time, is now calculated more reliably, and the UI will report 'waiting for a work slot' on pending jobs. no more blank pending!
+* when you pause mid-job, the 'pausing - status' text is generated is a little neater too
+* with luck, we'll also have fewer examples of 64KB of 503 error html spamming the UI
+* any critical unhandled errors during importing proper now stop that queue until a client restart and make an appropriate status text and popup (in some situations, they previously could spam every thirty seconds)
+* the simple downloader and urls downloader now support the 'delay work until later' error system. actual UI for status reporting on these downloaders remains limited, however
+* a bunch of misc downloader page cleanup
+
+### archive/delete
+* the final 'commit/forget/back' confirmation dialog on the archive/delete filter now lists all the possible local file domains you could delete from with separate file counts and 'commit' buttons, including 'all my files' if there are multiple, defaulting to the parent page's location at the top of the list. this let's you do a 'yes, purge all these from everywhere' delete or a 'no, just from here' delete as needed and generally makes what is going on more visible
+* fixed archive/delete commit for users with the 'archived file delete lock' turned on
+
+### misc
+* fixed a bug in the parsing sanity check that makes sure bad 'last modified' timestamps are not added. some ~1970-01-01 results were slipping through. on update, all modified dates within a week of this epoch will be retroactively removed
+* the 'connection' panel in the options now lets you configure how many times a network request can retry connections and requests. the logic behind these values is improved, too--network jobs now count connection and request errors separately
+* optimised the master tag update routine when you petition tags
+* the Client API help for /add_tags/add_tags now clarifies that deleting a tag that does not exist _will_ make a change--it makes a deletion record
+* thanks to a user, the 'getting started with files' help has had a pass
+* I looked into memory bloat some users are seeing after media viewer use, but I couldn't reproduce it locally. I am now making a plan to finally integrate a memory profiler and add some memory debug UI so we can better see what is going on when a couple gigs suddenly appear
+
+### important repository processing fixes
+* I've been trying to chase down a persistent processing bug some users got, where no matter what resyncs or checks they do, a content update seems to be cast as a definition update. fingers crossed, I have finally fixed it this week. it turns out there was a bug near my 'is this a definition or a content update?' check that is used for auto-repair maintenance here (long story short, ffmpeg was false-positive discovering mpegs in json). whatever the case, I have scheduled all users for a repository update file metadata check, so with luck anyone with a bad record will be fixed automatically in the background within a few hours of background work. anyone who encounters this problem in future should be fixed by the automatic repair too. thank you very much to the patient users who sent in reports about this and worked with me to figure this out. please try processing again, and let me know if you still have any issues
+* I also cleaned some of the maintenance code, and made it more aggressive, so 'do a full metadata resync' is now be even more uncompromising
+* also, the repository updates file service gets a bit of cleanup. it seems some ghost files have snuck in there over time, and today their records are corrected. the bug that let this happen in the first place is also fixed
+* there remains an issue where some users' clients have tried to hit the PTR with 404ing update file hashes. I am still investigating this
+
+## [Version 488](https://github.com/hydrusnetwork/hydrus/releases/tag/v488)
### all misc this week
* the client now supports 'wavpack' files. these are basically a kind of compressed wav. mpv seems to play them fine too!
@@ -311,38 +340,3 @@
* cleaned up an old wx label patch
* cleaned up an old wx system colour patch
* cleaned up some misc initialisation code
-
-## [Version 478](https://github.com/hydrusnetwork/hydrus/releases/tag/v478)
-
-### misc
-* if a file note text is crazy and can't be displayed, this is now handled and the best visual approximation is displayed (and saved back on ok) instead
-* fixed an error in the cloudflare problem detection calls for the newer versions of cloudscraper (>=1.2.60) while maintaining support for the older versions. fingers crossed, we also shouldn't repeat this specific error if they refactor again
-
-### file history chart updates
-* fixed the 'inbox' line in file history, which has to be calculated in an odd way and was not counting on file imports adding to the inbox
-* the file history chart now expands its y axis range to show all data even if deleted_files is huge. we'll see how nice this actually is IRL
-* bumped the file history resolution up from 1,000 to 2,000 steps
-* the y axis _should_ now show localised numbers, 5,000 instead of 5000, but the method by which this occurs involves fox tongues and the breath of a slighted widow, so it may just not work for some machines
-
-### cleanup, mostly file location stuff
-* I believe I have replaced all the remaining surplus static 'my files' references with code compatible with multiple local file services. when I add the capability to create new local file services, there now won't be a problem trying to display thumbnails or generate menu actions etc... if they aren't in 'my files'
-* pulled the autocomplete dropdown file domain button code out to its own class and refactored it and the multiple location context panel to their own file
-* added a 'default file location' option to 'files and trash' page, and a bunch of dialogs (e.g. the search panel when you make a new export folder) and similar now pull it to initialise. for most users this will stay 'my files' forever, but when we hit multiple local file services, it may want to change
-* the file domain override options in 'manage tag display and search' now work on the new location system and support multple file services
-* in downloaders, when highlighting, a database job that does the 'show files' filter (e.g. to include those in trash or not) now works on the new location context system and will handle files that will be imported to places other than my files
-* refactored client api file service parsing
-* refactored client api hashes parsing
-* cleaned a whole heap of misc location code
-* cleaned misc basic code across hydrus and client constant files
-* gave 'you don't want the server' help page a very quick pass
-
-### client api
-* in prep for multiple local file services, delete_files now takes an optional file_service_key or file_service_name. by default, it now deletes from all appropriate local services, so behaviour is unchanged from before without the parameter if you just want to delete m8
-* undelete files is the same. when we have multiple local file services, an undelete without a file service will undelete to all locations that have a delete record
-* delete_files also now takes an optional 'reason' parameter
-* the 'set_notes' command now checks the type of the notes Object. it obviously has to be string-to-string
-* the 'get_thumbnail' command should now never 404. if you ask for a pdf thumb, it gives the pdf default thumb, and if there is no thumb for whatever reason, you get the hydrus fallback thumbnail. just like in the client itself
-* updated client api help to talk about these
-* updated the unit tests to handle them too
-* did a pass over the client api help to unify indent style and fix other small formatting issues
-* client api version is now 28
diff --git a/docs/developer_api.md b/docs/developer_api.md
index 28233cde..668fb594 100644
--- a/docs/developer_api.md
+++ b/docs/developer_api.md
@@ -647,7 +647,14 @@ Response description:
: 200 and no content.
!!! note
- Note also that hydrus tag actions are safely idempotent. You can pend a tag that is already pended and not worry about an error--it will be discarded. The same for other reasonable logical scenarios: deleting a tag that does not exist will silently make no change, pending a tag that is already 'current' will again be passed over. It is fine to just throw 'process this' tags at every file import you add and not have to worry about checking which files you already added it to.
+ Note also that hydrus tag actions are safely idempotent. You can pend a tag that is already pended, or add a tag that already exists, and not worry about an error--the surplus add action will be discarded. The same is true if you try to pend a tag that actually already exists, or rescinding a petition that doesn't. Any invalid actions will fail silently.
+
+ It is fine to just throw your 'process this' tags at every file import and not have to worry about checking which files you already added them to.
+
+!!! danger "HOWEVER"
+ When you delete a tag, a deletion record is made _even if the tag does not exist on the file_. This is important if you expect to add the tags again via parsing, because, in general, when hydrus adds tags through a downloader, it will not overwrite a previously 'deleted' tag record (this is to stop re-downloads overwriting the tags you hand-removed previously). Undeletes usually have to be done manually by a human.
+
+ So, _do_ be careful about how you spam delete unless it is something that doesn't matter or it is something you'll only be touching again via the API anyway.
## Adding URLs
diff --git a/docs/old_changelog.html b/docs/old_changelog.html
index a4c3483a..2a6385ee 100644
--- a/docs/old_changelog.html
+++ b/docs/old_changelog.html
@@ -33,6 +33,35 @@
+
+
+ - downloader pages:
+ - greatly improved the status reporting for downloader pages. the way the little text updates on your file and gallery progress are generated and presented is overhauled, and tests are unified across the different downloader pages. you now get specific texts on all possible reasons the queue cannot currently process, such as the emergency pause states under the _network_ menu or specific info like hitting the file limit, and all the code involved here is much cleaner
+ - the 'working/pending' status, when you have a whole bunch of galleries or watchers wanting to run at the same time, is now calculated more reliably, and the UI will report 'waiting for a work slot' on pending jobs. no more blank pending!
+ - when you pause mid-job, the 'pausing - status' text is generated is a little neater too
+ - with luck, we'll also have fewer examples of 64KB of 503 error html spamming the UI
+ - any critical unhandled errors during importing proper now stop that queue until a client restart and make an appropriate status text and popup (in some situations, they previously could spam every thirty seconds)
+ - the simple downloader and urls downloader now support the 'delay work until later' error system. actual UI for status reporting on these downloaders remains limited, however
+ - a bunch of misc downloader page cleanup
+ - .
+ - archive/delete:
+ - the final 'commit/forget/back' confirmation dialog on the archive/delete filter now lists all the possible local file domains you could delete from with separate file counts and 'commit' buttons, including 'all my files' if there are multiple, defaulting to the parent page's location at the top of the list. this let's you do a 'yes, purge all these from everywhere' delete or a 'no, just from here' delete as needed and generally makes what is going on more visible
+ - fixed archive/delete commit for users with the 'archived file delete lock' turned on
+ - .
+ - misc:
+ - fixed a bug in the parsing sanity check that makes sure bad 'last modified' timestamps are not added. some ~1970-01-01 results were slipping through. on update, all modified dates within a week of this epoch will be retroactively removed
+ - the 'connection' panel in the options now lets you configure how many times a network request can retry connections and requests. the logic behind these values is improved, too--network jobs now count connection and request errors separately
+ - optimised the master tag update routine when you petition tags
+ - the Client API help for /add_tags/add_tags now clarifies that deleting a tag that does not exist _will_ make a change--it makes a deletion record
+ - thanks to a user, the 'getting started with files' help has had a pass
+ - I looked into memory bloat some users are seeing after media viewer use, but I couldn't reproduce it locally. I am now making a plan to finally integrate a memory profiler and add some memory debug UI so we can better see what is going on when a couple gigs suddenly appear
+ - .
+ - important repository processing fixes:
+ - I've been trying to chase down a persistent processing bug some users got, where no matter what resyncs or checks they do, a content update seems to be cast as a definition update. fingers crossed, I have finally fixed it this week. it turns out there was a bug near my 'is this a definition or a content update?' check that is used for auto-repair maintenance here (long story short, ffmpeg was false-positive discovering mpegs in json). whatever the case, I have scheduled all users for a repository update file metadata check, so with luck anyone with a bad record will be fixed automatically in the background within a few hours of background work. anyone who encounters this problem in future should be fixed by the automatic repair too. thank you very much to the patient users who sent in reports about this and worked with me to figure this out. please try processing again, and let me know if you still have any issues
+ - I also cleaned some of the maintenance code, and made it more aggressive, so 'do a full metadata resync' is now be even more uncompromising
+ - also, the repository updates file service gets a bit of cleanup. it seems some ghost files have snuck in there over time, and today their records are corrected. the bug that let this happen in the first place is also fixed
+ - there remains an issue where some users' clients have tried to hit the PTR with 404ing update file hashes. I am still investigating this
+
- the client now supports 'wavpack' files. these are basically a kind of compressed wav. mpv seems to play them fine too!
diff --git a/hydrus/client/ClientOptions.py b/hydrus/client/ClientOptions.py
index 901b5701..d3d48074 100644
--- a/hydrus/client/ClientOptions.py
+++ b/hydrus/client/ClientOptions.py
@@ -361,6 +361,9 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
self._dictionary[ 'integers' ][ 'max_network_jobs' ] = 15
self._dictionary[ 'integers' ][ 'max_network_jobs_per_domain' ] = 3
+ self._dictionary[ 'integers' ][ 'max_connection_attempts_allowed' ] = 5
+ self._dictionary[ 'integers' ][ 'max_request_attempts_allowed_get' ] = 5
+
from hydrus.core import HydrusImageHandling
self._dictionary[ 'integers' ][ 'thumbnail_scale_type' ] = HydrusImageHandling.THUMBNAIL_SCALE_DOWN_ONLY
diff --git a/hydrus/client/ClientTime.py b/hydrus/client/ClientTime.py
index 08e7f8c9..d3df830f 100644
--- a/hydrus/client/ClientTime.py
+++ b/hydrus/client/ClientTime.py
@@ -1,15 +1,17 @@
import typing
-from hydrus.core import HydrusData
-
-def ShouldUpdateDomainModifiedTime( existing_timestamp: int, new_timestamp: int ):
+def ShouldUpdateDomainModifiedTime( existing_timestamp: int, new_timestamp: typing.Optional[ int ] ) -> bool:
- # assume anything too early is a meme and a timestamp parsing conversion error
- if new_timestamp <= 86400 * 7:
+ if not TimestampIsSensible( new_timestamp ):
return False
+ if not TimestampIsSensible( existing_timestamp ):
+
+ return True
+
+
# only go backwards, in general
if new_timestamp >= existing_timestamp:
@@ -20,14 +22,14 @@ def ShouldUpdateDomainModifiedTime( existing_timestamp: int, new_timestamp: int
def MergeModifiedTimes( existing_timestamp: typing.Optional[ int ], new_timestamp: typing.Optional[ int ] ) -> typing.Optional[ int ]:
- if existing_timestamp is None:
+ if not TimestampIsSensible( existing_timestamp ):
- return new_timestamp
+ existing_timestamp = None
- if new_timestamp is None:
+ if not TimestampIsSensible( new_timestamp ):
- return existing_timestamp
+ new_timestamp = None
if ShouldUpdateDomainModifiedTime( existing_timestamp, new_timestamp ):
@@ -39,3 +41,19 @@ def MergeModifiedTimes( existing_timestamp: typing.Optional[ int ], new_timestam
return existing_timestamp
+
+def TimestampIsSensible( timestamp: typing.Optional[ int ] ) -> bool:
+
+ if timestamp is None:
+
+ return False
+
+
+ # assume anything too early is a meme and a timestamp parsing conversion error
+ if timestamp <= 86400 * 7:
+
+ return False
+
+
+ return True
+
diff --git a/hydrus/client/db/ClientDB.py b/hydrus/client/db/ClientDB.py
index a60249be..45335a41 100644
--- a/hydrus/client/db/ClientDB.py
+++ b/hydrus/client/db/ClientDB.py
@@ -262,6 +262,11 @@ class DB( HydrusDB.HydrusDB ):
self._AddFiles( self.modules_services.combined_local_file_service_id, valid_rows )
+ if service_type == HC.LOCAL_FILE_UPDATE_DOMAIN:
+
+ self._AddFiles( self.modules_services.combined_local_file_service_id, valid_rows )
+
+
# insert the files
pending_changed = self.modules_files_storage.AddFiles( service_id, valid_rows )
@@ -2349,7 +2354,12 @@ class DB( HydrusDB.HydrusDB ):
culled_mappings_ids = []
- for ( tag_id, hash_ids ) in mappings_ids:
+ for row in mappings_ids:
+
+ # mappings_ids here can have 'reason_id' for petitions, so we'll index our values here
+
+ tag_id = row[0]
+ hash_ids = row[1]
if len( hash_ids ) == 0:
@@ -2420,6 +2430,36 @@ class DB( HydrusDB.HydrusDB ):
valid_hash_ids = hash_ids
+ elif action == HC.CONTENT_UPDATE_PETITION:
+
+ result = self._Execute( 'SELECT 1 FROM {} WHERE tag_id = ? AND hash_id = ?;'.format( petitioned_mappings_table_name ), ( tag_id, hash_id ) ).fetchone()
+
+ if result is None:
+
+ valid_hash_ids = hash_ids
+
+ else:
+
+ continue
+
+
+ elif action == HC.CONTENT_UPDATE_RESCIND_PETITION:
+
+ result = self._Execute( 'SELECT 1 FROM {} WHERE tag_id = ? AND hash_id = ?;'.format( petitioned_mappings_table_name ), ( tag_id, hash_id ) ).fetchone()
+
+ if result is None:
+
+ continue
+
+ else:
+
+ valid_hash_ids = hash_ids
+
+
+ else:
+
+ valid_hash_ids = set()
+
else:
@@ -2439,7 +2479,9 @@ class DB( HydrusDB.HydrusDB ):
elif action == HC.CONTENT_UPDATE_PEND:
+ # prohibited hash_ids
existing_hash_ids = self._STS( self._Execute( 'SELECT hash_id FROM {} CROSS JOIN {} USING ( hash_id ) WHERE tag_id = ?;'.format( temp_hash_ids_table_name, current_mappings_table_name ), ( tag_id, ) ) )
+ # existing_hash_ids
existing_hash_ids.update( self._STI( self._Execute( 'SELECT hash_id FROM {} CROSS JOIN {} USING ( hash_id ) WHERE tag_id = ?;'.format( temp_hash_ids_table_name, pending_mappings_table_name ), ( tag_id, ) ) ) )
valid_hash_ids = set( hash_ids ).difference( existing_hash_ids )
@@ -2448,12 +2490,36 @@ class DB( HydrusDB.HydrusDB ):
valid_hash_ids = self._STS( self._Execute( 'SELECT hash_id FROM {} CROSS JOIN {} USING ( hash_id ) WHERE tag_id = ?;'.format( temp_hash_ids_table_name, pending_mappings_table_name ), ( tag_id, ) ) )
+ elif action == HC.CONTENT_UPDATE_PETITION:
+
+ # we are technically ok with deleting tags that don't exist yet!
+ existing_hash_ids = self._STS( self._Execute( 'SELECT hash_id FROM {} CROSS JOIN {} USING ( hash_id ) WHERE tag_id = ?;'.format( temp_hash_ids_table_name, petitioned_mappings_table_name ), ( tag_id, ) ) )
+
+ valid_hash_ids = set( hash_ids ).difference( existing_hash_ids )
+
+ elif action == HC.CONTENT_UPDATE_RESCIND_PETITION:
+
+ valid_hash_ids = self._STS( self._Execute( 'SELECT hash_id FROM {} CROSS JOIN {} USING ( hash_id ) WHERE tag_id = ?;'.format( temp_hash_ids_table_name, petitioned_mappings_table_name ), ( tag_id, ) ) )
+
+ else:
+
+ valid_hash_ids = set()
+
if len( valid_hash_ids ) > 0:
- culled_mappings_ids.append( ( tag_id, valid_hash_ids ) )
+ if action == HC.CONTENT_UPDATE_PETITION:
+
+ reason_id = row[2]
+
+ culled_mappings_ids.append( ( tag_id, valid_hash_ids, reason_id ) )
+
+ else:
+
+ culled_mappings_ids.append( ( tag_id, valid_hash_ids ) )
+
@@ -11872,6 +11938,36 @@ class DB( HydrusDB.HydrusDB ):
+ if version == 488:
+
+ # clearing up some garbo 1970-01-01 timestamps that got saved
+ self._Execute( 'DELETE FROM file_domain_modified_timestamps WHERE file_modified_timestamp < ?;', ( 86400 * 7, ) )
+
+ #
+
+ # mysterious situation where repo updates domain had some ghost files that were not in all local files!
+
+ hash_ids_in_repo_updates = set( self.modules_files_storage.GetCurrentHashIdsList( self.modules_services.local_update_service_id ) )
+
+ hash_ids_in_all_files = self.modules_files_storage.FilterHashIds( ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_FILE_SERVICE_KEY ), hash_ids_in_repo_updates )
+
+ orphan_hash_ids = hash_ids_in_repo_updates.difference( hash_ids_in_all_files )
+
+ if len( orphan_hash_ids ) > 0:
+
+ hash_ids_to_timestamps = self.modules_files_storage.GetCurrentHashIdsToTimestamps( self.modules_services.local_update_service_id, orphan_hash_ids )
+
+ rows = list( hash_ids_to_timestamps.items() )
+
+ self.modules_files_storage.AddFiles( self.modules_services.combined_local_file_service_id, rows )
+
+
+ # turns out ffmpeg was detecting some updates as mpegs, so this wasn't always working right!
+ self.modules_files_maintenance_queue.AddJobs( hash_ids_in_repo_updates, ClientFiles.REGENERATE_FILE_DATA_JOB_FILE_METADATA )
+
+ self._Execute( 'DELETE FROM service_info WHERE service_id = ?;', ( self.modules_services.local_update_service_id, ) )
+
+
self._controller.frame_splash_status.SetTitleText( 'updated db to v{}'.format( HydrusData.ToHumanInt( version + 1 ) ) )
self._Execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
@@ -11892,6 +11988,8 @@ class DB( HydrusDB.HydrusDB ):
deleted_mappings_ids = self._FilterExistingUpdateMappings( tag_service_id, deleted_mappings_ids, HC.CONTENT_UPDATE_DELETE )
pending_mappings_ids = self._FilterExistingUpdateMappings( tag_service_id, pending_mappings_ids, HC.CONTENT_UPDATE_PEND )
pending_rescinded_mappings_ids = self._FilterExistingUpdateMappings( tag_service_id, pending_rescinded_mappings_ids, HC.CONTENT_UPDATE_RESCIND_PEND )
+ petitioned_mappings_ids = self._FilterExistingUpdateMappings( tag_service_id, petitioned_mappings_ids, HC.CONTENT_UPDATE_PETITION )
+ petitioned_rescinded_mappings_ids = self._FilterExistingUpdateMappings( tag_service_id, petitioned_rescinded_mappings_ids, HC.CONTENT_UPDATE_RESCIND_PETITION )
tag_ids_to_filter_chained = { tag_id for ( tag_id, hash_ids ) in itertools.chain.from_iterable( ( mappings_ids, deleted_mappings_ids, pending_mappings_ids, pending_rescinded_mappings_ids ) ) }
diff --git a/hydrus/client/db/ClientDBRepositories.py b/hydrus/client/db/ClientDBRepositories.py
index bd5cef36..27c7a8a0 100644
--- a/hydrus/client/db/ClientDBRepositories.py
+++ b/hydrus/client/db/ClientDBRepositories.py
@@ -166,7 +166,9 @@ class ClientDBRepositories( ClientDBModule.ClientDBModule ):
raise Exception( message )
- def _RegisterUpdates( self, service_id, hash_ids = None ):
+ def _RegisterLocalUpdates( self, service_id, hash_ids = None ):
+
+ # this function takes anything in 'unregistered', sees what is local, and figures out the correct 'content types' for those hash ids in the 'processed' table. converting unknown/bad hash_ids to correct and ready to process
# it is ok if this guy gets hash ids that are already in the 'processed' table--it'll now resync them and correct if needed
@@ -224,7 +226,7 @@ class ClientDBRepositories( ClientDBModule.ClientDBModule ):
if len( insert_rows ) > 0:
processed = False
-
+
self._ExecuteMany( 'INSERT OR IGNORE INTO {} ( hash_id, content_type, processed ) VALUES ( ?, ?, ? );'.format( repository_updates_processed_table_name ), ( ( hash_id, content_type, processed ) for ( hash_id, content_type ) in insert_rows ) )
@@ -287,7 +289,7 @@ class ClientDBRepositories( ClientDBModule.ClientDBModule ):
self._ExecuteMany( 'INSERT OR IGNORE INTO {} ( hash_id ) VALUES ( ? );'.format( repository_unregistered_updates_table_name ), ( ( hash_id, ) for ( update_index, hash_id ) in inserts ) )
- self._RegisterUpdates( service_id )
+ self._RegisterLocalUpdates( service_id )
def DropRepositoryTables( self, service_id: int ):
@@ -310,7 +312,7 @@ class ClientDBRepositories( ClientDBModule.ClientDBModule ):
for service_id in self.modules_services.GetServiceIds( HC.REPOSITORIES ):
- self._RegisterUpdates( service_id )
+ self._RegisterLocalUpdates( service_id )
@@ -594,13 +596,15 @@ class ClientDBRepositories( ClientDBModule.ClientDBModule ):
def NotifyUpdatesChanged( self, hash_ids ):
+ # a mime changed
+
for service_id in self.modules_services.GetServiceIds( HC.REPOSITORIES ):
( repository_updates_table_name, repository_unregistered_updates_table_name, repository_updates_processed_table_name ) = GenerateRepositoryUpdatesTableNames( service_id )
self._ExecuteMany( 'INSERT OR IGNORE INTO {} ( hash_id ) VALUES ( ? );'.format( repository_unregistered_updates_table_name ), ( ( hash_id, ) for hash_id in hash_ids ) )
- self._RegisterUpdates( service_id, hash_ids )
+ self._RegisterLocalUpdates( service_id, hash_ids )
@@ -608,7 +612,7 @@ class ClientDBRepositories( ClientDBModule.ClientDBModule ):
for service_id in self.modules_services.GetServiceIds( HC.REPOSITORIES ):
- self._RegisterUpdates( service_id, hash_ids )
+ self._RegisterLocalUpdates( service_id, hash_ids )
@@ -722,19 +726,17 @@ class ClientDBRepositories( ClientDBModule.ClientDBModule ):
current_update_hash_ids = self._STS( self._Execute( 'SELECT hash_id FROM {};'.format( repository_updates_table_name ) ) )
- deletee_hash_ids = current_update_hash_ids.difference( all_future_update_hash_ids )
-
- self._ExecuteMany( 'DELETE FROM {} WHERE hash_id = ?;'.format( repository_updates_table_name ), ( ( hash_id, ) for hash_id in deletee_hash_ids ) )
+ self._Execute( 'DELETE FROM {};'.format( repository_updates_table_name ) )
#
self._Execute( 'DELETE FROM {};'.format( repository_unregistered_updates_table_name ) )
- #
+ # we want to keep 'yes we processed this' records on a full metadata resync
good_current_hash_ids = current_update_hash_ids.intersection( all_future_update_hash_ids )
- current_processed_table_update_hash_ids = self._STS( self._Execute( 'SELECT hash_id FROM {};'.format( repository_updates_processed_table_name ) ) )
+ current_processed_table_update_hash_ids = self._STS( self._Execute( 'SELECT hash_id FROM {} WHERE processed = ?;'.format( repository_updates_processed_table_name ), ( True, ) ) )
deletee_processed_table_update_hash_ids = current_processed_table_update_hash_ids.difference( good_current_hash_ids )
@@ -750,21 +752,14 @@ class ClientDBRepositories( ClientDBModule.ClientDBModule ):
hash_id = self.modules_hashes_local_cache.GetHashId( update_hash )
- if hash_id in current_update_hash_ids:
-
- self._Execute( 'UPDATE {} SET update_index = ? WHERE hash_id = ?;'.format( repository_updates_table_name ), ( update_index, hash_id ) )
-
- else:
-
- inserts.append( ( update_index, hash_id ) )
-
+ inserts.append( ( update_index, hash_id ) )
self._ExecuteMany( 'INSERT OR IGNORE INTO {} ( update_index, hash_id ) VALUES ( ?, ? );'.format( repository_updates_table_name ), inserts )
self._ExecuteMany( 'INSERT OR IGNORE INTO {} ( hash_id ) VALUES ( ? );'.format( repository_unregistered_updates_table_name ), ( ( hash_id, ) for hash_id in all_future_update_hash_ids ) )
- self._RegisterUpdates( service_id )
+ self._RegisterLocalUpdates( service_id )
def SetUpdateProcessed( self, service_id: int, update_hash: bytes, content_types: typing.Collection[ int ] ):
@@ -780,4 +775,3 @@ class ClientDBRepositories( ClientDBModule.ClientDBModule ):
self._ClearOutstandingWorkCache( service_id, content_type )
-
\ No newline at end of file
diff --git a/hydrus/client/gui/ClientGUI.py b/hydrus/client/gui/ClientGUI.py
index 0eedca7d..8b6f68d6 100644
--- a/hydrus/client/gui/ClientGUI.py
+++ b/hydrus/client/gui/ClientGUI.py
@@ -3313,7 +3313,7 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes, CAC.ApplicationCo
ClientGUIMenus.AppendMenuItem( memory_actions, 'run fast memory maintenance', 'Tell all the fast caches to maintain themselves.', self._controller.MaintainMemoryFast )
ClientGUIMenus.AppendMenuItem( memory_actions, 'run slow memory maintenance', 'Tell all the slow caches to maintain themselves.', self._controller.MaintainMemorySlow )
- ClientGUIMenus.AppendMenuItem( memory_actions, 'clear image rendering cache', 'Tell the image rendering system to forget all current images. This will often free up a bunch of memory immediately.', self._controller.ClearCaches )
+ ClientGUIMenus.AppendMenuItem( memory_actions, 'clear all rendering caches', 'Tell the image rendering system to forget all current images, tiles, and thumbs. This will often free up a bunch of memory immediately.', self._controller.ClearCaches )
ClientGUIMenus.AppendMenuItem( memory_actions, 'clear thumbnail cache', 'Tell the thumbnail cache to forget everything and redraw all current thumbs.', self._controller.pub, 'reset_thumbnail_cache' )
ClientGUIMenus.AppendMenuItem( memory_actions, 'print garbage', 'Print some information about the python garbage to the log.', self._DebugPrintGarbage )
ClientGUIMenus.AppendMenuItem( memory_actions, 'take garbage snapshot', 'Capture current garbage object counts.', self._DebugTakeGarbageSnapshot )
diff --git a/hydrus/client/gui/ClientGUIDialogsQuick.py b/hydrus/client/gui/ClientGUIDialogsQuick.py
index bde44862..83107866 100644
--- a/hydrus/client/gui/ClientGUIDialogsQuick.py
+++ b/hydrus/client/gui/ClientGUIDialogsQuick.py
@@ -37,6 +37,21 @@ def GetDeleteFilesJobs( win, media, default_reason, suggested_file_service_key =
+def GetFinishArchiveDeleteFilteringAnswer( win, kept_label, deletion_options ):
+
+ with ClientGUITopLevelWindowsPanels.DialogCustomButtonQuestion( win, 'filtering done?' ) as dlg:
+
+ panel = ClientGUIScrolledPanelsButtonQuestions.QuestionArchiveDeleteFinishFilteringPanel( dlg, kept_label, deletion_options )
+
+ dlg.SetPanel( panel )
+
+ result = dlg.exec()
+ location_context = panel.GetLocationContext()
+ was_cancelled = dlg.WasCancelled()
+
+ return ( result, location_context, was_cancelled )
+
+
def GetFinishFilteringAnswer( win, label ):
with ClientGUITopLevelWindowsPanels.DialogCustomButtonQuestion( win, label ) as dlg:
diff --git a/hydrus/client/gui/ClientGUIExport.py b/hydrus/client/gui/ClientGUIExport.py
index c3bcb71e..e859ba78 100644
--- a/hydrus/client/gui/ClientGUIExport.py
+++ b/hydrus/client/gui/ClientGUIExport.py
@@ -234,7 +234,7 @@ class EditExportFolderPanel( ClientGUIScrolledPanels.EditPanel ):
self._query_box = ClientGUICommon.StaticBox( self, 'query to export' )
- self._page_key = 'export folders placeholder'
+ self._page_key = b'export folders placeholder'
self._tag_autocomplete = ClientGUIACDropdown.AutoCompleteDropdownTagsRead( self._query_box, self._page_key, file_search_context, allow_all_known_files = False, force_system_everything = True )
diff --git a/hydrus/client/gui/ClientGUIImport.py b/hydrus/client/gui/ClientGUIImport.py
index 17adafa1..b2abc301 100644
--- a/hydrus/client/gui/ClientGUIImport.py
+++ b/hydrus/client/gui/ClientGUIImport.py
@@ -1952,32 +1952,8 @@ class GalleryImportPanel( ClientGUICommon.StaticBox ):
ClientGUIFunctions.SetBitmapButtonBitmap( self._gallery_pause_button, CC.global_pixmaps().gallery_pause )
- if gallery_paused:
-
- if gallery_status == '':
-
- gallery_status = 'paused'
-
- else:
-
- gallery_status = 'paused - ' + gallery_status
-
-
-
self._gallery_status.setText( gallery_status )
- if files_paused:
-
- if file_status == '':
-
- file_status = 'paused'
-
- else:
-
- file_status = 'pausing - ' + file_status
-
-
-
self._file_status.setText( file_status )
( file_network_job, gallery_network_job ) = self._gallery_import.GetNetworkJobs()
@@ -2554,15 +2530,6 @@ class WatcherReviewPanel( ClientGUICommon.StaticBox ):
if files_paused:
- if file_status == '':
-
- file_status = 'paused'
-
- else:
-
- file_status = 'pausing, ' + file_status
-
-
ClientGUIFunctions.SetBitmapButtonBitmap( self._files_pause_button, CC.global_pixmaps().file_play )
else:
@@ -2576,11 +2543,6 @@ class WatcherReviewPanel( ClientGUICommon.StaticBox ):
if checking_paused:
- if watcher_status == '':
-
- watcher_status = 'paused'
-
-
ClientGUIFunctions.SetBitmapButtonBitmap( self._checking_pause_button, CC.global_pixmaps().gallery_play )
else:
diff --git a/hydrus/client/gui/ClientGUIScrolledPanelsButtonQuestions.py b/hydrus/client/gui/ClientGUIScrolledPanelsButtonQuestions.py
index 2b27e853..b7c14f28 100644
--- a/hydrus/client/gui/ClientGUIScrolledPanelsButtonQuestions.py
+++ b/hydrus/client/gui/ClientGUIScrolledPanelsButtonQuestions.py
@@ -1,9 +1,13 @@
+import os
+import typing
+
from qtpy import QtCore as QC
from qtpy import QtWidgets as QW
from hydrus.core import HydrusGlobals as HG
from hydrus.client import ClientConstants as CC
+from hydrus.client import ClientLocation
from hydrus.client.gui import ClientGUIFunctions
from hydrus.client.gui import ClientGUIScrolledPanels
from hydrus.client.gui import QtPorting as QP
@@ -41,6 +45,133 @@ class QuestionCommitInterstitialFilteringPanel( ClientGUIScrolledPanels.Resizing
ClientGUIFunctions.SetFocusLater( self._commit )
+class QuestionArchiveDeleteFinishFilteringPanel( ClientGUIScrolledPanels.ResizingScrolledPanel ):
+
+ def __init__( self, parent, kept_label: typing.Optional[ str ], deletion_options ):
+
+ ClientGUIScrolledPanels.ResizingScrolledPanel.__init__( self, parent )
+
+ self._location_context = ClientLocation.LocationContext() # empty
+
+ vbox = QP.VBoxLayout()
+
+ first_commit = None
+
+ if len( deletion_options ) == 0:
+
+ if kept_label is None:
+
+ kept_label = 'ERROR: do not seem to have any actions at all!'
+
+
+ label = '{}?'.format( kept_label )
+
+ st = ClientGUICommon.BetterStaticText( self, label )
+
+ st.setAlignment( QC.Qt.AlignCenter )
+
+ QP.AddToLayout( vbox, st, CC.FLAGS_EXPAND_PERPENDICULAR )
+
+ first_commit = ClientGUICommon.BetterButton( self, 'commit', self.DoCommit, ClientLocation.LocationContext() )
+ first_commit.setObjectName( 'HydrusAccept' )
+
+ QP.AddToLayout( vbox, first_commit, CC.FLAGS_EXPAND_PERPENDICULAR )
+
+ elif len( deletion_options ) == 1:
+
+ ( location_context, delete_label ) = deletion_options[0]
+
+ if kept_label is None:
+
+ label = '{}?'.format( delete_label )
+
+ else:
+
+ label = '{} and {}?'.format( kept_label, delete_label )
+
+
+ st = ClientGUICommon.BetterStaticText( self, label )
+
+ st.setAlignment( QC.Qt.AlignCenter )
+
+ QP.AddToLayout( vbox, st, CC.FLAGS_EXPAND_PERPENDICULAR )
+
+ first_commit = ClientGUICommon.BetterButton( self, 'commit', self.DoCommit, location_context )
+ first_commit.setObjectName( 'HydrusAccept' )
+
+ QP.AddToLayout( vbox, first_commit, CC.FLAGS_EXPAND_PERPENDICULAR )
+
+ else:
+
+ if kept_label is not None:
+
+ label = '{}{}-and-'.format( kept_label, os.linesep )
+
+ st = ClientGUICommon.BetterStaticText( self, label )
+
+ st.setAlignment( QC.Qt.AlignCenter )
+
+ QP.AddToLayout( vbox, st, CC.FLAGS_EXPAND_PERPENDICULAR )
+
+
+ for ( location_context, delete_label ) in deletion_options:
+
+ label = '{}?'.format( delete_label )
+
+ st = ClientGUICommon.BetterStaticText( self, label )
+
+ st.setAlignment( QC.Qt.AlignCenter )
+
+ QP.AddToLayout( vbox, st, CC.FLAGS_EXPAND_PERPENDICULAR )
+
+ commit = ClientGUICommon.BetterButton( self, 'commit', self.DoCommit, location_context )
+ commit.setObjectName( 'HydrusAccept' )
+
+ QP.AddToLayout( vbox, commit, CC.FLAGS_EXPAND_PERPENDICULAR )
+
+ if first_commit is None:
+
+ first_commit = commit
+
+
+
+
+ self._forget = ClientGUICommon.BetterButton( self, 'forget', self.parentWidget().done, QW.QDialog.Rejected )
+ self._forget.setObjectName( 'HydrusCancel' )
+
+ self._back = ClientGUICommon.BetterButton( self, 'back to filtering', self.DoGoBack )
+
+ QP.AddToLayout( vbox, self._forget, CC.FLAGS_EXPAND_PERPENDICULAR )
+
+ st = ClientGUICommon.BetterStaticText( self, '-or-' )
+
+ st.setAlignment( QC.Qt.AlignCenter )
+
+ QP.AddToLayout( vbox, st, CC.FLAGS_EXPAND_PERPENDICULAR )
+ QP.AddToLayout( vbox, self._back, CC.FLAGS_EXPAND_PERPENDICULAR )
+
+ self.widget().setLayout( vbox )
+
+ ClientGUIFunctions.SetFocusLater( first_commit )
+
+
+ def DoGoBack( self ):
+
+ self.parentWidget().SetCancelled( True )
+ self.parentWidget().done( QW.QDialog.Rejected )
+
+
+ def DoCommit( self, location_context ):
+
+ self._location_context = location_context
+ self.parentWidget().done( QW.QDialog.Accepted )
+
+
+ def GetLocationContext( self ) -> ClientLocation.LocationContext:
+
+ return self._location_context
+
+
class QuestionFinishFilteringPanel( ClientGUIScrolledPanels.ResizingScrolledPanel ):
def __init__( self, parent, label ):
diff --git a/hydrus/client/gui/ClientGUIScrolledPanelsManagement.py b/hydrus/client/gui/ClientGUIScrolledPanelsManagement.py
index 2ae04d5b..5dcdf656 100644
--- a/hydrus/client/gui/ClientGUIScrolledPanelsManagement.py
+++ b/hydrus/client/gui/ClientGUIScrolledPanelsManagement.py
@@ -304,8 +304,14 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
max_network_jobs_per_domain_max = 5
+ self._max_connection_attempts_allowed = ClientGUICommon.BetterSpinBox( general, min = 1, max = 10 )
+ self._max_connection_attempts_allowed.setToolTip( 'This refers to timeouts when actually making the initial connection.' )
+
+ self._max_request_attempts_allowed_get = ClientGUICommon.BetterSpinBox( general, min = 1, max = 10 )
+ self._max_request_attempts_allowed_get.setToolTip( 'This refers to timeouts when waiting for a response to our GET requests, whether that is the start or an interruption part way through.' )
+
self._network_timeout = ClientGUICommon.BetterSpinBox( general, min = network_timeout_min, max = network_timeout_max )
- self._network_timeout.setToolTip( 'If a network connection cannot be made in this duration or, if once started, it experiences uninterrupted inactivity for six times this duration, it will be abandoned.' )
+ self._network_timeout.setToolTip( 'If a network connection cannot be made in this duration or, once started, it experiences inactivity for six times this duration, it will be considered dead and retried or abandoned.' )
self._connection_error_wait_time = ClientGUICommon.BetterSpinBox( general, min = error_wait_time_min, max = error_wait_time_max )
self._connection_error_wait_time.setToolTip( 'If a network connection times out as above, it will wait increasing multiples of this base time before retrying.' )
@@ -334,6 +340,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._https_proxy.SetValue( self._new_options.GetNoneableString( 'https_proxy' ) )
self._no_proxy.SetValue( self._new_options.GetNoneableString( 'no_proxy' ) )
+ self._max_connection_attempts_allowed.setValue( self._new_options.GetInteger( 'max_connection_attempts_allowed' ) )
+ self._max_request_attempts_allowed_get.setValue( self._new_options.GetInteger( 'max_request_attempts_allowed_get' ) )
self._network_timeout.setValue( self._new_options.GetInteger( 'network_timeout' ) )
self._connection_error_wait_time.setValue( self._new_options.GetInteger( 'connection_error_wait_time' ) )
self._serverside_bandwidth_wait_time.setValue( self._new_options.GetInteger( 'serverside_bandwidth_wait_time' ) )
@@ -362,6 +370,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
rows = []
+ rows.append( ( 'max connection attempts allowed per request: ', self._max_connection_attempts_allowed ) )
+ rows.append( ( 'max retries allowed per request: ', self._max_request_attempts_allowed_get ) )
rows.append( ( 'network timeout (seconds): ', self._network_timeout ) )
rows.append( ( 'connection error retry wait (seconds): ', self._connection_error_wait_time ) )
rows.append( ( 'serverside bandwidth retry wait (seconds): ', self._serverside_bandwidth_wait_time ) )
@@ -425,7 +435,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._new_options.SetNoneableString( 'https_proxy', self._https_proxy.GetValue() )
self._new_options.SetNoneableString( 'no_proxy', self._no_proxy.GetValue() )
- self._new_options.SetInteger( 'network_timeout', self._network_timeout.value() )
+ self._new_options.SetInteger( 'max_connection_attempts_allowed', self._max_connection_attempts_allowed.value() )
+ self._new_options.SetInteger( 'max_request_attempts_allowed_get', self._max_request_attempts_allowed_get.value() )
self._new_options.SetInteger( 'connection_error_wait_time', self._connection_error_wait_time.value() )
self._new_options.SetInteger( 'serverside_bandwidth_wait_time', self._serverside_bandwidth_wait_time.value() )
self._new_options.SetInteger( 'max_network_jobs', self._max_network_jobs.value() )
diff --git a/hydrus/client/gui/canvas/ClientGUICanvas.py b/hydrus/client/gui/canvas/ClientGUICanvas.py
index 4467dcb3..39891a3f 100644
--- a/hydrus/client/gui/canvas/ClientGUICanvas.py
+++ b/hydrus/client/gui/canvas/ClientGUICanvas.py
@@ -4224,7 +4224,7 @@ class CanvasMediaListFilterArchiveDelete( CanvasMediaList ):
if delete_lock_for_archived_files:
- deleted = [ media.GetHash() for media in self._deleted if not media.HasArchive() ]
+ deleted = [ media for media in self._deleted if not media.HasArchive() ]
else:
@@ -4233,33 +4233,70 @@ class CanvasMediaListFilterArchiveDelete( CanvasMediaList ):
if len( kept ) > 0 or len( deleted ) > 0:
- label_components = []
-
if len( kept ) > 0:
- label_components.append( 'keep {}'.format( HydrusData.ToHumanInt( len( kept ) ) ) )
+ kept_label = 'keep {}'.format( HydrusData.ToHumanInt( len( kept ) ) )
+ else:
+
+ kept_label = None
+
+
+ deletion_options = []
if len( deleted ) > 0:
- if self._location_context.IncludesCurrent():
+ location_contexts_to_present_options_for = [ self._location_context ]
+
+ current_local_service_keys = HydrusData.MassUnion( [ m.GetLocationsManager().GetCurrent() for m in deleted ] )
+
+ local_file_domain_service_keys = [ service_key for service_key in current_local_service_keys if HG.client_controller.services_manager.GetServiceType( service_key ) == HC.LOCAL_FILE_DOMAIN ]
+
+ location_contexts_to_present_options_for.extend( [ ClientLocation.LocationContext.STATICCreateSimple( service_key ) for service_key in local_file_domain_service_keys ] )
+
+ if len( local_file_domain_service_keys ) > 1:
- location_context_label = self._location_context.ToString( HG.client_controller.services_manager.GetName )
-
- else:
-
- location_context_label = 'all possible local file services'
+ location_contexts_to_present_options_for.append( ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY ) )
- label_components.append( 'delete (from {}) {}'.format( location_context_label, HydrusData.ToHumanInt( len( self._deleted ) ) ) )
+ specific_local_service_keys = [ service_key for service_key in current_local_service_keys if HG.client_controller.services_manager.GetServiceType( service_key ) in HC.SPECIFIC_LOCAL_FILE_SERVICES ]
+
+ if len( specific_local_service_keys ) > len( local_file_domain_service_keys ): # we have some trash or I guess repo updates
+
+ location_contexts_to_present_options_for.append( ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_FILE_SERVICE_KEY ) )
+
+
+ location_contexts_to_present_options_for = HydrusData.DedupeList( location_contexts_to_present_options_for )
+
+ for location_context in location_contexts_to_present_options_for:
+
+ file_service_keys = location_context.current_service_keys
+
+ num_deletable = len( [ m for m in deleted if len( m.GetLocationsManager().GetCurrent().intersection( file_service_keys ) ) > 0 ] )
+
+ if num_deletable > 0:
+
+ if location_context == ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY ):
+
+ location_label = 'all local file domains'
+
+ elif location_context == ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_FILE_SERVICE_KEY ):
+
+ location_label = 'my hard disk'
+
+ else:
+
+ location_label = location_context.ToString( HG.client_controller.services_manager.GetName )
+
+
+ delete_label = 'delete {} from {}'.format( HydrusData.ToHumanInt( num_deletable ), location_label )
+
+ deletion_options.append( ( location_context, delete_label ) )
+
+
- label = '{} files?'.format( ' and '.join( label_components ) )
-
- # TODO: ok so ideally we should total up the deleteds' actual file services and give users UI to select what they want to delete from
- # so like '23 files in my files, 2 in favourites' and then a 'yes' button for all or just my files or favourites
-
- ( result, cancelled ) = ClientGUIDialogsQuick.GetFinishFilteringAnswer( self, label )
+ ( result, deletee_location_context, cancelled ) = ClientGUIDialogsQuick.GetFinishArchiveDeleteFilteringAnswer( self, kept_label, deletion_options )
if cancelled:
@@ -4282,7 +4319,7 @@ class CanvasMediaListFilterArchiveDelete( CanvasMediaList ):
self._current_media = self._GetFirst() # so the pubsub on close is better
- HG.client_controller.CallToThread( CommitArchiveDelete, self._page_key, self._location_context, kept, deleted )
+ HG.client_controller.CallToThread( CommitArchiveDelete, self._page_key, deletee_location_context, kept, deleted )
diff --git a/hydrus/client/gui/pages/ClientGUIManagement.py b/hydrus/client/gui/pages/ClientGUIManagement.py
index 39ca5231..16285816 100644
--- a/hydrus/client/gui/pages/ClientGUIManagement.py
+++ b/hydrus/client/gui/pages/ClientGUIManagement.py
@@ -1830,18 +1830,6 @@ class ManagementPanelImporterHDD( ManagementPanelImporter ):
ClientGUIFunctions.SetBitmapButtonBitmap( self._pause_button, CC.global_pixmaps().file_pause )
- if paused:
-
- if current_action == '':
-
- current_action = 'paused'
-
- else:
-
- current_action = 'pausing - ' + current_action
-
-
-
self._current_action.setText( current_action )
@@ -3862,18 +3850,8 @@ class ManagementPanelImporterSimpleDownloader( ManagementPanelImporter ):
self._pending_jobs_listbox.SelectData( selected_jobs )
- if queue_paused:
-
- parser_status = 'paused'
-
-
self._parser_status.setText( parser_status )
- if current_action == '' and files_paused:
-
- current_action = 'paused'
-
-
self._current_action.setText( current_action )
if files_paused:
diff --git a/hydrus/client/gui/pages/ClientGUIResults.py b/hydrus/client/gui/pages/ClientGUIResults.py
index 40b46bf5..2ec17a7c 100644
--- a/hydrus/client/gui/pages/ClientGUIResults.py
+++ b/hydrus/client/gui/pages/ClientGUIResults.py
@@ -2142,7 +2142,7 @@ class MediaPanel( ClientMedia.ListeningMediaList, QW.QScrollArea, CAC.Applicatio
def ClearPageKey( self ):
- self._page_key = 'dead media panel page key'
+ self._page_key = b'dead media panel page key'
def Collect( self, page_key, media_collect = None ):
diff --git a/hydrus/client/importing/ClientImportControl.py b/hydrus/client/importing/ClientImportControl.py
new file mode 100644
index 00000000..085014f7
--- /dev/null
+++ b/hydrus/client/importing/ClientImportControl.py
@@ -0,0 +1,129 @@
+import typing
+
+from hydrus.core import HydrusData
+from hydrus.core import HydrusExceptions
+from hydrus.core import HydrusGlobals as HG
+
+from hydrus.client.importing import ClientImportFileSeeds
+from hydrus.client.importing import ClientImportGallerySeeds
+from hydrus.client.importing.options import FileImportOptions
+
+def CheckImporterCanDoFileWorkBecausePaused( paused: bool, file_seed_cache: ClientImportFileSeeds.FileSeedCache, page_key: bytes ):
+
+ if paused:
+
+ raise HydrusExceptions.VetoException( 'paused' )
+
+
+ if HG.client_controller.new_options.GetBoolean( 'pause_all_file_queues' ):
+
+ raise HydrusExceptions.VetoException( 'all file import queues are paused!' )
+
+
+ work_pending = file_seed_cache.WorkToDo()
+
+ if not work_pending:
+
+ raise HydrusExceptions.VetoException()
+
+
+ if HG.client_controller.PageClosedButNotDestroyed( page_key ):
+
+ raise HydrusExceptions.VetoException( 'page is closed' )
+
+
+
+def CheckImporterCanDoFileWorkBecausePausifyingProblem( file_import_options: FileImportOptions ):
+
+ try:
+
+ file_import_options.CheckReadyToImport()
+
+ except Exception as e:
+
+ raise HydrusExceptions.VetoException( str( e ) )
+
+
+
+def CheckImporterCanDoGalleryWorkBecausePaused( paused: bool, gallery_seed_log: typing.Optional[ ClientImportGallerySeeds.GallerySeedLog ] ):
+
+ if paused:
+
+ raise HydrusExceptions.VetoException( 'paused' )
+
+
+ if HG.client_controller.new_options.GetBoolean( 'pause_all_gallery_searches' ):
+
+ raise HydrusExceptions.VetoException( 'all gallery searches are paused!' )
+
+
+ if gallery_seed_log is not None:
+
+ work_pending = gallery_seed_log.WorkToDo()
+
+ if not work_pending:
+
+ raise HydrusExceptions.VetoException()
+
+
+
+
+def CheckCanDoNetworkWork( no_work_until: int, no_work_until_reason: str ):
+
+ if not HydrusData.TimeHasPassed( no_work_until ):
+
+ no_work_text = '{}: {}'.format( HydrusData.ConvertTimestampToPrettyExpires( no_work_until ), no_work_until_reason )
+
+ raise HydrusExceptions.VetoException( no_work_text )
+
+
+ if HG.client_controller.network_engine.IsBusy():
+
+ raise HydrusExceptions.VetoException( 'network engine is too busy!' )
+
+
+
+def CheckImporterCanDoWorkBecauseStopped( page_key: bytes ):
+
+ if PageImporterShouldStopWorking( page_key ):
+
+ raise HydrusExceptions.VetoException( 'page should stop working' )
+
+
+
+def GenerateLiveStatusText( text: str, paused: bool, no_work_until: int, no_work_until_reason: str ) -> str:
+
+ if not HydrusData.TimeHasPassed( no_work_until ):
+
+ return '{}: {}'.format( HydrusData.ConvertTimestampToPrettyExpires( no_work_until ), no_work_until_reason )
+
+
+ if paused and text != 'paused':
+
+ if text == '':
+
+ text = 'pausing'
+
+ else:
+
+ text = 'pausing - {}'.format( text )
+
+
+
+ return text
+
+
+def NeatenStatusText( text: str ) -> str:
+
+ if len( text ) > 0:
+
+ text = text.splitlines()[0]
+
+
+ return text
+
+
+def PageImporterShouldStopWorking( page_key: bytes ):
+
+ return HG.started_shutdown or not HG.client_controller.PageAlive( page_key )
+
diff --git a/hydrus/client/importing/ClientImportFileSeeds.py b/hydrus/client/importing/ClientImportFileSeeds.py
index 1ece8494..a057f4cf 100644
--- a/hydrus/client/importing/ClientImportFileSeeds.py
+++ b/hydrus/client/importing/ClientImportFileSeeds.py
@@ -364,7 +364,7 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
source_timestamp = min( HydrusData.GetNow() - 30, source_timestamp )
- self.source_time = source_timestamp
+ self.source_time = ClientTime.MergeModifiedTimes( self.source_time, source_timestamp )
self._UpdateModified()
diff --git a/hydrus/client/importing/ClientImportGallery.py b/hydrus/client/importing/ClientImportGallery.py
index 511f2b46..0942459f 100644
--- a/hydrus/client/importing/ClientImportGallery.py
+++ b/hydrus/client/importing/ClientImportGallery.py
@@ -12,8 +12,8 @@ from hydrus.client import ClientDownloading
from hydrus.client.importing import ClientImportFileSeeds
from hydrus.client.importing import ClientImportGallerySeeds
from hydrus.client.importing import ClientImporting
+from hydrus.client.importing import ClientImportControl
from hydrus.client.importing.options import FileImportOptions
-from hydrus.client.importing.options import PresentationImportOptions
from hydrus.client.importing.options import TagImportOptions
from hydrus.client.networking import ClientNetworkingJobs
@@ -51,7 +51,7 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
self._source_name = source_name
- self._page_key = 'initialising page key'
+ self._page_key = b'initialising page key'
self._publish_to_page = False
self._current_page_index = 0
@@ -79,11 +79,15 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
self._no_work_until_reason = ''
self._lock = threading.Lock()
+ self._files_working_lock = threading.Lock()
+ self._gallery_working_lock = threading.Lock()
- self._file_status = ''
+ self._files_status = ''
self._gallery_status = ''
self._gallery_status_can_change_timestamp = 0
+ self._we_are_probably_pending = True
+
self._all_work_finished = False
self._have_started = False
@@ -112,6 +116,11 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
def _DelayWork( self, time_delta, reason ):
+ if len( reason ) > 0:
+
+ reason = reason.splitlines()[0]
+
+
self._no_work_until = HydrusData.GetNow() + time_delta
self._no_work_until_reason = reason
@@ -229,12 +238,7 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- if len( text ) > 0:
-
- text = text.splitlines()[0]
-
-
- self._file_status = text
+ self._files_status = ClientImportControl.NeatenStatusText( text )
@@ -309,6 +313,14 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
return
+ def status_hook( text ):
+
+ with self._lock:
+
+ self._gallery_status = ClientImportControl.NeatenStatusText( text )
+
+
+
with self._lock:
if self._AmOverFileLimit():
@@ -318,8 +330,8 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
return
- self._gallery_status = 'checking next page'
-
+
+ status_hook( 'checking next page' )
def file_seeds_callable( file_seeds ):
@@ -335,19 +347,6 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
return ClientImporting.UpdateFileSeedCacheWithFileSeeds( self._file_seed_cache, file_seeds, max_new_urls_allowed = max_new_urls_allowed )
- def status_hook( text ):
-
- with self._lock:
-
- if len( text ) > 0:
-
- text = text.splitlines()[0]
-
-
- self._gallery_status = text
-
-
-
def title_hook( text ):
return
@@ -524,25 +523,6 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
- def GetGalleryStatus( self ):
-
- with self._lock:
-
- if HydrusData.TimeHasPassed( self._no_work_until ):
-
- gallery_status = self._gallery_status
-
- else:
-
- no_work_text = '{}: {}'.format( HydrusData.ConvertTimestampToPrettyExpires( self._no_work_until ), self._no_work_until_reason )
-
- gallery_status = no_work_text
-
-
- return gallery_status
-
-
-
def GetHashes( self ):
with self._lock:
@@ -614,13 +594,16 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_DONE, 'DONE' )
- elif self._gallery_status != '' or self._file_status != '':
-
- return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_WORKING, 'working' )
-
elif gallery_go or files_go:
- return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_PENDING, 'pending' )
+ if self._gallery_working_lock.locked() or self._files_working_lock.locked():
+
+ return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_WORKING, 'working' )
+
+ else:
+
+ return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_PENDING, 'pending' )
+
else:
@@ -639,22 +622,31 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
def GetStatus( self ):
+ # first off, if we are waiting for a work slot, our status is not being updated!
+ # so, we'll update it here as needed
+
with self._lock:
- if HydrusData.TimeHasPassed( self._no_work_until ):
+ gallery_work_to_do = self._gallery_seed_log.WorkToDo()
+ files_work_to_do = self._file_seed_cache.WorkToDo()
+
+ gallery_go = gallery_work_to_do and not self._gallery_paused
+ files_go = files_work_to_do and not self._files_paused
+
+ if gallery_go and not self._gallery_working_lock.locked():
- gallery_status = self._gallery_status
- file_status = self._file_status
-
- else:
-
- no_work_text = '{}: {}'.format( HydrusData.ConvertTimestampToPrettyExpires( self._no_work_until ), self._no_work_until_reason )
-
- gallery_status = no_work_text
- file_status = no_work_text
+ self._gallery_status = 'waiting for a work slot'
- return ( gallery_status, file_status, self._files_paused, self._gallery_paused )
+ if files_go and not self._files_working_lock.locked():
+
+ self._files_status = 'waiting for a work slot'
+
+
+ gallery_text = ClientImportControl.GenerateLiveStatusText( self._gallery_status, self._gallery_paused, self._no_work_until, self._no_work_until_reason )
+ file_text = ClientImportControl.GenerateLiveStatusText( self._files_status, self._files_paused, self._no_work_until, self._no_work_until_reason )
+
+ return ( gallery_text, file_text, self._files_paused, self._gallery_paused )
@@ -846,74 +838,43 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
- def CanDoFileWork( self ):
+ def CheckCanDoFileWork( self ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( self._page_key ):
-
- self._files_repeating_job.Cancel()
-
- return False
-
-
- files_paused = self._files_paused or HG.client_controller.new_options.GetBoolean( 'pause_all_file_queues' )
-
- if files_paused:
-
- return False
-
-
try:
- self._file_import_options.CheckReadyToImport()
+ ClientImportControl.CheckImporterCanDoWorkBecauseStopped( self._page_key )
- except Exception as e:
+ except HydrusExceptions.VetoException:
- self._file_status = str( e )
+ self._files_repeating_job.Cancel()
- HydrusData.ShowText( str( e ) )
+ raise
+
+
+ ClientImportControl.CheckImporterCanDoFileWorkBecausePaused( self._files_paused, self._file_seed_cache, self._page_key )
+
+ try:
+
+ ClientImportControl.CheckImporterCanDoFileWorkBecausePausifyingProblem( self._file_import_options )
+
+ except HydrusExceptions.VetoException:
self._files_paused = True
- return False
-
-
- work_pending = self._file_seed_cache.WorkToDo()
-
- if not work_pending:
-
- return False
+ raise
- return self.CanDoNetworkWork()
+ self.CheckCanDoNetworkWork()
- def CanDoNetworkWork( self ):
+ def CheckCanDoNetworkWork( self ):
with self._lock:
- no_delays = HydrusData.TimeHasPassed( self._no_work_until )
-
- if not no_delays:
-
- return False
-
-
- page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
-
- if not page_shown:
-
- return False
-
-
- network_engine_good = not HG.client_controller.network_engine.IsBusy()
-
- if not network_engine_good:
-
- return False
-
+ ClientImportControl.CheckCanDoNetworkWork( self._no_work_until, self._no_work_until_reason )
return True
@@ -921,12 +882,26 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
def REPEATINGWorkOnFiles( self ):
- try:
+ with self._files_working_lock:
- while self.CanDoFileWork():
+ while True:
try:
+ try:
+
+ self.CheckCanDoFileWork()
+
+ except HydrusExceptions.VetoException as e:
+
+ with self._lock:
+
+ self._files_status = str( e )
+
+
+ break
+
+
self._WorkOnFiles()
HG.client_controller.WaitUntilViewFree()
@@ -935,56 +910,67 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
except Exception as e:
+ with self._lock:
+
+ self._files_status = 'stopping work: {}'.format( str( e ) )
+
+
HydrusData.ShowException( e )
-
-
- finally:
-
- with self._lock:
-
- self._file_status = ''
+ return
+
- def CanDoGalleryWork( self ):
+ def CheckCanDoGalleryWork( self ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( self._page_key ):
+ try:
+
+ ClientImportControl.CheckImporterCanDoWorkBecauseStopped( self._page_key )
+
+ except HydrusExceptions.VetoException:
self._gallery_repeating_job.Cancel()
- return False
+ raise
- gallery_paused = self._gallery_paused or HG.client_controller.new_options.GetBoolean( 'pause_all_gallery_searches' )
-
- if gallery_paused:
+ if self._AmOverFileLimit():
- return False
+ raise HydrusExceptions.VetoException( 'hit file limit' )
- work_pending = self._gallery_seed_log.WorkToDo()
-
- if not work_pending:
-
- return False
-
+ ClientImportControl.CheckImporterCanDoGalleryWorkBecausePaused( self._gallery_paused, self._gallery_seed_log )
- return self.CanDoNetworkWork()
+ return self.CheckCanDoNetworkWork()
def REPEATINGWorkOnGallery( self ):
- try:
+ with self._gallery_working_lock:
- while self.CanDoGalleryWork():
+ while True:
try:
+ try:
+
+ self.CheckCanDoGalleryWork()
+
+ except HydrusExceptions.VetoException as e:
+
+ with self._lock:
+
+ self._gallery_status = str( e )
+
+
+ break
+
+
self._WorkOnGallery()
time.sleep( 1 )
@@ -995,19 +981,20 @@ class GalleryImport( HydrusSerialisable.SerialisableBase ):
except Exception as e:
+ with self._lock:
+
+ self._gallery_status = 'stopping work: {}'.format( str( e ) )
+
+
HydrusData.ShowException( e )
-
-
- finally:
-
- with self._lock:
-
- self._gallery_status = ''
+ return
+
+
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_GALLERY_IMPORT ] = GalleryImport
class MultipleGalleryImport( HydrusSerialisable.SerialisableBase ):
@@ -1027,7 +1014,7 @@ class MultipleGalleryImport( HydrusSerialisable.SerialisableBase ):
self._lock = threading.Lock()
- self._page_key = 'initialising page key'
+ self._page_key = b'initialising page key'
self._gug_key_and_name = gug_key_and_name
@@ -1737,7 +1724,7 @@ class MultipleGalleryImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( self._page_key ):
+ if ClientImportControl.PageImporterShouldStopWorking( self._page_key ):
self._importers_repeating_job.Cancel()
diff --git a/hydrus/client/importing/ClientImportLocal.py b/hydrus/client/importing/ClientImportLocal.py
index ae7e8c11..98a089dd 100644
--- a/hydrus/client/importing/ClientImportLocal.py
+++ b/hydrus/client/importing/ClientImportLocal.py
@@ -4,6 +4,7 @@ import time
from hydrus.core import HydrusConstants as HC
from hydrus.core import HydrusData
+from hydrus.core import HydrusExceptions
from hydrus.core import HydrusFileHandling
from hydrus.core import HydrusGlobals as HG
from hydrus.core import HydrusPaths
@@ -15,6 +16,7 @@ from hydrus.client import ClientData
from hydrus.client import ClientFiles
from hydrus.client import ClientPaths
from hydrus.client import ClientThreading
+from hydrus.client.importing import ClientImportControl
from hydrus.client.importing import ClientImporting
from hydrus.client.importing import ClientImportFileSeeds
from hydrus.client.importing.options import FileImportOptions
@@ -70,7 +72,9 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
self._file_import_options = file_import_options
self._delete_after_success = delete_after_success
- self._current_action = ''
+ self._page_key = b'initialising page key'
+
+ self._files_status = ''
self._paused = False
self._lock = threading.Lock()
@@ -131,7 +135,7 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
- def _WorkOnFiles( self, page_key ):
+ def _WorkOnFiles( self ):
file_seed = self._file_seed_cache.GetNextFileSeed( CC.STATUS_UNKNOWN )
@@ -140,39 +144,28 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
return
- did_substantial_work = False
-
path = file_seed.file_seed_data
with self._lock:
- self._current_action = 'importing'
+ self._files_status = 'importing'
def status_hook( text ):
with self._lock:
- if len( text ) > 0:
-
- text = text.splitlines()[0]
-
-
- self._current_action = text
+ self._file_status = ClientImportControl.NeatenStatusText( text )
file_seed.ImportPath( self._file_seed_cache, self._file_import_options, status_hook = status_hook )
- did_substantial_work = True
-
if file_seed.status in CC.SUCCESSFUL_IMPORT_STATES:
if file_seed.ShouldPresent( self._file_import_options.GetPresentationImportOptions() ):
- file_seed.PresentToPage( page_key )
-
- did_substantial_work = True
+ file_seed.PresentToPage( self._page_key )
if self._delete_after_success:
@@ -206,13 +199,10 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- self._current_action = ''
+ self._files_status = ''
- if did_substantial_work:
-
- time.sleep( ClientImporting.DID_SUBSTANTIAL_FILE_WORK_MINIMUM_SLEEP_TIME )
-
+ time.sleep( ClientImporting.DID_SUBSTANTIAL_FILE_WORK_MINIMUM_SLEEP_TIME )
def CurrentlyWorking( self ):
@@ -264,7 +254,9 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- return ( self._current_action, self._paused )
+ text = ClientImportControl.GenerateLiveStatusText( self._files_status, self._paused, 0, '' )
+
+ return ( text, self._paused )
@@ -321,69 +313,55 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
def Start( self, page_key ):
- self._files_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnFiles, page_key )
+ self._page_key = page_key
+
+ self._files_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnFiles )
self._files_repeating_job.SetThreadSlotType( 'misc' )
- def CanDoFileWork( self, page_key ):
+ def CheckCanDoFileWork( self ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( page_key ):
+ try:
+
+ ClientImportControl.CheckImporterCanDoWorkBecauseStopped( self._page_key )
+
+ except HydrusExceptions.VetoException:
self._files_repeating_job.Cancel()
- return False
+ raise
- paused = self._paused or HG.client_controller.new_options.GetBoolean( 'pause_all_file_queues' )
-
- if paused:
-
- return False
-
-
- try:
-
- self._file_import_options.CheckReadyToImport()
-
- except Exception as e:
-
- self._current_action = str( e )
-
- HydrusData.ShowText( str( e ) )
-
- self._paused = True
-
- return False
-
-
- work_to_do = self._file_seed_cache.WorkToDo()
-
- if not work_to_do:
-
- return False
-
-
- page_shown = not HG.client_controller.PageClosedButNotDestroyed( page_key )
-
- if not page_shown:
-
- return False
-
+ ClientImportControl.CheckImporterCanDoFileWorkBecausePaused( self._paused, self._file_seed_cache, self._page_key )
return True
- def REPEATINGWorkOnFiles( self, page_key ):
+ def REPEATINGWorkOnFiles( self ):
- while self.CanDoFileWork( page_key ):
+ while True:
try:
- self._WorkOnFiles( page_key )
+ try:
+
+ self.CheckCanDoFileWork()
+
+ except HydrusExceptions.VetoException as e:
+
+ with self._lock:
+
+ self._file_status = str( e )
+
+
+ break
+
+
+ self._WorkOnFiles()
HG.client_controller.WaitUntilViewFree()
@@ -391,8 +369,15 @@ class HDDImport( HydrusSerialisable.SerialisableBase ):
except Exception as e:
+ with self._lock:
+
+ self._file_status = 'stopping work: {}'.format( str( e ) )
+
+
HydrusData.ShowException( e )
+ return
+
diff --git a/hydrus/client/importing/ClientImportSimpleURLs.py b/hydrus/client/importing/ClientImportSimpleURLs.py
index cc5f2f64..74dde7eb 100644
--- a/hydrus/client/importing/ClientImportSimpleURLs.py
+++ b/hydrus/client/importing/ClientImportSimpleURLs.py
@@ -1,4 +1,3 @@
-import os
import threading
import time
import urllib.parse
@@ -10,6 +9,7 @@ from hydrus.core import HydrusGlobals as HG
from hydrus.core import HydrusSerialisable
from hydrus.client import ClientConstants as CC
+from hydrus.client.importing import ClientImportControl
from hydrus.client.importing import ClientImporting
from hydrus.client.importing import ClientImportFileSeeds
from hydrus.client.importing import ClientImportGallerySeeds
@@ -35,13 +35,17 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
self._file_seed_cache = ClientImportFileSeeds.FileSeedCache()
self._file_import_options = file_import_options
self._formula_name = 'all files linked by images in page'
- self._queue_paused = False
+ self._gallery_paused = False
self._files_paused = False
+ self._no_work_until = 0
+ self._no_work_until_reason = ''
+
+ self._page_key = b'initialising page key'
self._downloader_key = HydrusData.GenerateKey()
- self._parser_status = ''
- self._current_action = ''
+ self._gallery_status = ''
+ self._files_status = ''
self._lock = threading.Lock()
@@ -51,13 +55,24 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
self._page_network_job = None
self._files_repeating_job = None
- self._queue_repeating_job = None
+ self._gallery_repeating_job = None
self._last_serialisable_change_timestamp = 0
HG.client_controller.sub( self, 'NotifyFileSeedsUpdated', 'file_seed_cache_file_seeds_updated' )
+ def _DelayWork( self, time_delta, reason ):
+
+ if len( reason ) > 0:
+
+ reason = reason.splitlines()[0]
+
+
+ self._no_work_until = HydrusData.GetNow() + time_delta
+ self._no_work_until_reason = reason
+
+
def _FileNetworkJobPresentationContextFactory( self, network_job ):
def enter_call():
@@ -87,12 +102,12 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
serialisable_file_seed_cache = self._file_seed_cache.GetSerialisableTuple()
serialisable_file_import_options = self._file_import_options.GetSerialisableTuple()
- return ( serialisable_pending_jobs, serialisable_gallery_seed_log, serialisable_file_seed_cache, serialisable_file_import_options, self._formula_name, self._queue_paused, self._files_paused )
+ return ( serialisable_pending_jobs, serialisable_gallery_seed_log, serialisable_file_seed_cache, serialisable_file_import_options, self._formula_name, self._gallery_paused, self._files_paused )
def _InitialiseFromSerialisableInfo( self, serialisable_info ):
- ( serialisable_pending_jobs, serialisable_gallery_seed_log, serialisable_file_seed_cache, serialisable_file_import_options, self._formula_name, self._queue_paused, self._files_paused ) = serialisable_info
+ ( serialisable_pending_jobs, serialisable_gallery_seed_log, serialisable_file_seed_cache, serialisable_file_import_options, self._formula_name, self._gallery_paused, self._files_paused ) = serialisable_info
self._pending_jobs = [ ( url, HydrusSerialisable.CreateFromSerialisableTuple( serialisable_simple_downloader_formula ) ) for ( url, serialisable_simple_downloader_formula ) in serialisable_pending_jobs ]
@@ -186,7 +201,7 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
- def _WorkOnFiles( self, page_key ):
+ def _WorkOnFiles( self ):
file_seed = self._file_seed_cache.GetNextFileSeed( CC.STATUS_UNKNOWN )
@@ -201,29 +216,45 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- if len( text ) > 0:
-
- text = text.splitlines()[0]
-
-
- self._current_action = text
+ self._files_status = ClientImportControl.NeatenStatusText( text )
tag_import_options = TagImportOptions.TagImportOptions( is_default = True )
- did_substantial_work = file_seed.WorkOnURL( self._file_seed_cache, status_hook, self._NetworkJobFactory, self._FileNetworkJobPresentationContextFactory, self._file_import_options, tag_import_options )
+ try:
+
+ did_substantial_work = file_seed.WorkOnURL( self._file_seed_cache, status_hook, self._NetworkJobFactory, self._FileNetworkJobPresentationContextFactory, self._file_import_options, tag_import_options )
+
+ except HydrusExceptions.NetworkException as e:
+
+ delay = HG.client_controller.new_options.GetInteger( 'downloader_network_error_delay' )
+
+ self._DelayWork( delay, str( e ) )
+
+ file_seed.SetStatus( CC.STATUS_ERROR, str( e ) )
+
+ HydrusData.PrintException( e )
+
+ except Exception as e:
+
+ status = CC.STATUS_ERROR
+
+ file_seed.SetStatus( status, exception = e )
+
+ time.sleep( 3 )
+
if file_seed.ShouldPresent( self._file_import_options.GetPresentationImportOptions() ):
- file_seed.PresentToPage( page_key )
+ file_seed.PresentToPage( self._page_key )
did_substantial_work = True
with self._lock:
- self._current_action = ''
+ self._files_status = ''
if did_substantial_work:
@@ -232,7 +263,7 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
- def _WorkOnQueue( self, page_key ):
+ def _WorkOnGallery( self ):
if len( self._pending_jobs ) > 0:
@@ -240,15 +271,18 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
( url, simple_downloader_formula ) = self._pending_jobs.pop( 0 )
- self._parser_status = 'checking ' + url
+ self._gallery_status = 'checking ' + url
error_occurred = False
+ gallery_seed_status = CC.STATUS_ERROR
+ parser_status = 'job not completed'
+
+ gallery_seed = ClientImportGallerySeeds.GallerySeed( url, can_generate_more_pages = False )
+
try:
- gallery_seed = ClientImportGallerySeeds.GallerySeed( url, can_generate_more_pages = False )
-
self._gallery_seed_log.AddGallerySeeds( ( gallery_seed, ) )
network_job = self._NetworkJobFactory( 'GET', url )
@@ -325,6 +359,19 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
parser_status = 'page 404'
+ except HydrusExceptions.NetworkException as e:
+
+ delay = HG.client_controller.new_options.GetInteger( 'downloader_network_error_delay' )
+
+ self._DelayWork( delay, str( e ) )
+
+ gallery_seed_status = CC.STATUS_ERROR
+ error_occurred = True
+
+ parser_status = str( e )
+
+ HydrusData.PrintException( e )
+
except Exception as e:
gallery_seed_status = CC.STATUS_ERROR
@@ -344,7 +391,7 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- self._parser_status = parser_status
+ self._gallery_status = ClientImportControl.NeatenStatusText( parser_status )
if error_occurred:
@@ -358,7 +405,7 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- self._parser_status = ''
+ self._gallery_status = ''
return False
@@ -440,7 +487,7 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
d[ 'files_paused' ] = self._files_paused
- d[ 'gallery_paused' ] = self._queue_paused
+ d[ 'gallery_paused' ] = self._gallery_paused
return d
@@ -498,7 +545,10 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- return ( list( self._pending_jobs ), self._parser_status, self._current_action, self._queue_paused, self._files_paused )
+ gallery_text = ClientImportControl.GenerateLiveStatusText( self._gallery_status, self._gallery_paused, self._no_work_until, self._no_work_until_reason )
+ file_text = ClientImportControl.GenerateLiveStatusText( self._files_status, self._files_paused, self._no_work_until, self._no_work_until_reason )
+
+ return ( list( self._pending_jobs ), gallery_text, file_text, self._gallery_paused, self._files_paused )
@@ -544,9 +594,9 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- self._queue_paused = not self._queue_paused
+ self._gallery_paused = not self._gallery_paused
- ClientImporting.WakeRepeatingJob( self._queue_repeating_job )
+ ClientImporting.WakeRepeatingJob( self._gallery_repeating_job )
self._SerialisableChangeMade()
@@ -560,7 +610,7 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
self._pending_jobs.append( job )
- ClientImporting.WakeRepeatingJob( self._queue_repeating_job )
+ ClientImporting.WakeRepeatingJob( self._gallery_repeating_job )
self._SerialisableChangeMade()
@@ -602,89 +652,81 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
return
- self._files_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnFiles, page_key )
- self._queue_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnQueue, page_key )
+ self._page_key = page_key
+
+ self._files_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnFiles )
+ self._gallery_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnGallery )
self._files_repeating_job.SetThreadSlotType( 'misc' )
- self._queue_repeating_job.SetThreadSlotType( 'misc' )
+ self._gallery_repeating_job.SetThreadSlotType( 'misc' )
self._have_started = True
- def CanDoFileWork( self, page_key ):
+ def CheckCanDoFileWork( self ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( page_key ):
-
- self._files_repeating_job.Cancel()
-
- return False
-
-
- files_paused = self._files_paused or HG.client_controller.new_options.GetBoolean( 'pause_all_file_queues' )
-
- if files_paused:
-
- return False
-
-
try:
- self._file_import_options.CheckReadyToImport()
+ ClientImportControl.CheckImporterCanDoWorkBecauseStopped( self._page_key )
- except Exception as e:
+ except HydrusExceptions.VetoException:
- self._current_action = str( e )
+ self._files_repeating_job.Cancel()
- HydrusData.ShowText( str( e ) )
+ raise
+
+
+ ClientImportControl.CheckImporterCanDoFileWorkBecausePaused( self._files_paused, self._file_seed_cache, self._page_key )
+
+ try:
+
+ ClientImportControl.CheckImporterCanDoFileWorkBecausePausifyingProblem( self._file_import_options )
+
+ except HydrusExceptions.VetoException:
self._files_paused = True
- return False
-
-
- work_to_do = self._file_seed_cache.WorkToDo()
-
- if not work_to_do:
-
- return False
+ raise
- return self.CanDoNetworkWork( page_key )
+ self.CheckCanDoNetworkWork()
- def CanDoNetworkWork( self, page_key ):
+ def CheckCanDoNetworkWork( self ):
with self._lock:
- page_shown = not HG.client_controller.PageClosedButNotDestroyed( page_key )
-
- if not page_shown:
-
- return False
-
-
- network_engine_good = not HG.client_controller.network_engine.IsBusy()
-
- if not network_engine_good:
-
- return False
-
+ ClientImportControl.CheckCanDoNetworkWork( self._no_work_until, self._no_work_until_reason )
return True
- def REPEATINGWorkOnFiles( self, page_key ):
+ def REPEATINGWorkOnFiles( self ):
- while self.CanDoFileWork( page_key ):
+ while True:
try:
- self._WorkOnFiles( page_key )
+ try:
+
+ self.CheckCanDoFileWork()
+
+ except HydrusExceptions.VetoException as e:
+
+ with self._lock:
+
+ self._files_status = str( e )
+
+
+ break
+
+
+ self._WorkOnFiles()
HG.client_controller.WaitUntilViewFree()
@@ -692,49 +734,67 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
except Exception as e:
+ with self._lock:
+
+ self._files_status = 'stopping work: {}'.format( str( e ) )
+
+
HydrusData.ShowException( e )
+ return
+
- def CanDoQueueWork( self, page_key ):
+ def CheckCanDoGalleryWork( self ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( page_key ):
+ try:
- self._queue_repeating_job.Cancel()
+ ClientImportControl.CheckImporterCanDoWorkBecauseStopped( self._page_key )
- return False
+ except HydrusExceptions.VetoException:
+
+ self._gallery_repeating_job.Cancel()
+
+ raise
- queue_paused = self._queue_paused or HG.client_controller.new_options.GetBoolean( 'pause_all_gallery_searches' )
+ if len( self._pending_jobs ) == 0:
+
+ raise HydrusExceptions.VetoException()
+
- if queue_paused:
-
- return False
-
+ ClientImportControl.CheckImporterCanDoGalleryWorkBecausePaused( self._gallery_paused, None )
- return self.CanDoNetworkWork( page_key )
+ return self.CheckCanDoNetworkWork()
- def REPEATINGWorkOnQueue( self, page_key ):
+ def REPEATINGWorkOnGallery( self ):
- while self.CanDoQueueWork( page_key ):
+ while True:
try:
- did_work = self._WorkOnQueue( page_key )
+ try:
+
+ self.CheckCanDoGalleryWork()
+
+ except HydrusExceptions.VetoException as e:
+
+ with self._lock:
+
+ self._gallery_status = str( e )
+
+
+ break
+
- if did_work:
-
- time.sleep( ClientImporting.DID_SUBSTANTIAL_FILE_WORK_MINIMUM_SLEEP_TIME )
-
- else:
-
- return
-
+ self._WorkOnGallery()
+
+ time.sleep( 1 )
HG.client_controller.WaitUntilViewFree()
@@ -742,11 +802,19 @@ class SimpleDownloaderImport( HydrusSerialisable.SerialisableBase ):
except Exception as e:
+ with self._lock:
+
+ self._gallery_status = 'stopping work: {}'.format( str( e ) )
+
+
HydrusData.ShowException( e )
+ return
+
+
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_SIMPLE_DOWNLOADER_IMPORT ] = SimpleDownloaderImport
class URLsImport( HydrusSerialisable.SerialisableBase ):
@@ -765,12 +833,19 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
self._tag_import_options = TagImportOptions.TagImportOptions( is_default = True )
self._paused = False
+ self._no_work_until = 0
+ self._no_work_until_reason = ''
+
+ self._page_key = b'initialising page key'
self._downloader_key = HydrusData.GenerateKey()
self._lock = threading.Lock()
self._have_started = False
+ self._files_status = ''
+ self._gallery_status = ''
+
self._files_network_job = None
self._gallery_network_job = None
@@ -783,6 +858,17 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
HG.client_controller.sub( self, 'NotifyGallerySeedsUpdated', 'gallery_seed_log_gallery_seeds_updated' )
+ def _DelayWork( self, time_delta, reason ):
+
+ if len( reason ) > 0:
+
+ reason = reason.splitlines()[0]
+
+
+ self._no_work_until = HydrusData.GetNow() + time_delta
+ self._no_work_until_reason = reason
+
+
def _FileNetworkJobPresentationContextFactory( self, network_job ):
def enter_call():
@@ -886,7 +972,7 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
- def _WorkOnFiles( self, page_key ):
+ def _WorkOnFiles( self ):
file_seed = self._file_seed_cache.GetNextFileSeed( CC.STATUS_UNKNOWN )
@@ -907,11 +993,21 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
if file_seed.ShouldPresent( self._file_import_options.GetPresentationImportOptions() ):
- file_seed.PresentToPage( page_key )
+ file_seed.PresentToPage( self._page_key )
did_substantial_work = True
+ except HydrusExceptions.NetworkException as e:
+
+ delay = HG.client_controller.new_options.GetInteger( 'downloader_network_error_delay' )
+
+ self._DelayWork( delay, str( e ) )
+
+ file_seed.SetStatus( CC.STATUS_ERROR, str( e ) )
+
+ HydrusData.PrintException( e )
+
except Exception as e:
status = CC.STATUS_ERROR
@@ -927,7 +1023,7 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
- def _WorkOnGallery( self, page_key ):
+ def _WorkOnGallery( self ):
gallery_seed = self._gallery_seed_log.GetNextGallerySeed( CC.STATUS_UNKNOWN )
@@ -948,6 +1044,16 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
gallery_seed.WorkOnURL( 'download page', self._gallery_seed_log, file_seeds_callable, status_hook, title_hook, self._NetworkJobFactory, self._GalleryNetworkJobPresentationContextFactory, self._file_import_options )
+ except HydrusExceptions.NetworkException as e:
+
+ delay = HG.client_controller.new_options.GetInteger( 'downloader_network_error_delay' )
+
+ self._DelayWork( delay, str( e ) )
+
+ gallery_seed.SetStatus( CC.STATUS_ERROR, str( e ) )
+
+ HydrusData.PrintException( e )
+
except Exception as e:
status = CC.STATUS_ERROR
@@ -1199,8 +1305,10 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
return
- self._files_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnFiles, page_key )
- self._gallery_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnGallery, page_key )
+ self._page_key = page_key
+
+ self._files_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnFiles )
+ self._gallery_repeating_job = HG.client_controller.CallRepeating( ClientImporting.GetRepeatingJobInitialDelay(), ClientImporting.REPEATING_JOB_TYPICAL_PERIOD, self.REPEATINGWorkOnGallery )
self._files_repeating_job.SetThreadSlotType( 'misc' )
self._gallery_repeating_job.SetThreadSlotType( 'misc' )
@@ -1209,124 +1317,132 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
- def CanDoFileWork( self, page_key ):
+ def CheckCanDoFileWork( self ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( page_key ):
-
- self._files_repeating_job.Cancel()
-
- return False
-
-
- files_paused = self._paused or HG.client_controller.new_options.GetBoolean( 'pause_all_file_queues' )
-
- if files_paused:
-
- return False
-
-
try:
- self._file_import_options.CheckReadyToImport()
+ ClientImportControl.CheckImporterCanDoWorkBecauseStopped( self._page_key )
- except Exception as e:
+ except HydrusExceptions.VetoException:
+
+ self._files_repeating_job.Cancel()
+
+ raise
+
+
+ ClientImportControl.CheckImporterCanDoFileWorkBecausePaused( self._paused, self._file_seed_cache, self._page_key )
+
+ try:
+
+ ClientImportControl.CheckImporterCanDoFileWorkBecausePausifyingProblem( self._file_import_options )
+
+ except HydrusExceptions.VetoException:
self._paused = True
- HydrusData.ShowText( str( e ) )
-
- return False
-
-
- work_to_do = self._file_seed_cache.WorkToDo()
-
- if not work_to_do:
-
- return False
+ raise
- return self.CanDoNetworkWork( page_key )
+ self.CheckCanDoNetworkWork()
- def CanDoNetworkWork( self, page_key ):
+ def CheckCanDoNetworkWork( self ):
with self._lock:
- page_shown = not HG.client_controller.PageClosedButNotDestroyed( page_key )
-
- if not page_shown:
-
- return False
-
-
- network_engine_good = not HG.client_controller.network_engine.IsBusy()
-
- if not network_engine_good:
-
- return False
-
+ ClientImportControl.CheckCanDoNetworkWork( self._no_work_until, self._no_work_until_reason )
return True
- def REPEATINGWorkOnFiles( self, page_key ):
+ def REPEATINGWorkOnFiles( self ):
- while self.CanDoFileWork( page_key ):
+ while True:
try:
- self._WorkOnFiles( page_key )
+ try:
+
+ self.CheckCanDoFileWork()
+
+ except HydrusExceptions.VetoException as e:
+
+ with self._lock:
+
+ self._files_status = str( e )
+
+
+ break
+
+
+ self._WorkOnFiles()
HG.client_controller.WaitUntilViewFree()
+
self._SerialisableChangeMade()
except Exception as e:
+ with self._lock:
+
+ self._files_status = 'stopping work: {}'.format( str( e ) )
+
+
HydrusData.ShowException( e )
+ return
+
- def CanDoGalleryWork( self, page_key ):
+ def CheckCanDoGalleryWork( self ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( page_key ):
+ try:
+
+ ClientImportControl.CheckImporterCanDoWorkBecauseStopped( self._page_key )
+
+ except HydrusExceptions.VetoException:
self._gallery_repeating_job.Cancel()
- return False
+ raise
- gallery_paused = self._paused or HG.client_controller.new_options.GetBoolean( 'pause_all_gallery_searches' )
-
- if gallery_paused:
-
- return False
-
-
- work_to_do = self._gallery_seed_log.WorkToDo()
-
- if not work_to_do:
-
- return False
-
+ ClientImportControl.CheckImporterCanDoGalleryWorkBecausePaused( self._paused, self._gallery_seed_log )
- return self.CanDoNetworkWork( page_key )
+ return self.CheckCanDoNetworkWork()
- def REPEATINGWorkOnGallery( self, page_key ):
+ def REPEATINGWorkOnGallery( self ):
- while self.CanDoGalleryWork( page_key ):
+ while True:
try:
- self._WorkOnGallery( page_key )
+ try:
+
+ self.CheckCanDoGalleryWork()
+
+ except HydrusExceptions.VetoException as e:
+
+ with self._lock:
+
+ self._gallery_status = str( e )
+
+
+ break
+
+
+ self._WorkOnGallery()
+
+ time.sleep( 1 )
HG.client_controller.WaitUntilViewFree()
@@ -1334,9 +1450,17 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
except Exception as e:
+ with self._lock:
+
+ self._gallery_status = 'stopping work: {}'.format( str( e ) )
+
+
HydrusData.ShowException( e )
+ return
+
+
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_URLS_IMPORT ] = URLsImport
diff --git a/hydrus/client/importing/ClientImportSubscriptionLegacy.py b/hydrus/client/importing/ClientImportSubscriptionLegacy.py
index af83215a..50f868f8 100644
--- a/hydrus/client/importing/ClientImportSubscriptionLegacy.py
+++ b/hydrus/client/importing/ClientImportSubscriptionLegacy.py
@@ -541,6 +541,11 @@ class SubscriptionLegacy( HydrusSerialisable.SerialisableBaseNamed ):
def _DelayWork( self, time_delta, reason ):
+ if len( reason ) > 0:
+
+ reason = reason.splitlines()[0]
+
+
self._no_work_until = HydrusData.GetNow() + time_delta
self._no_work_until_reason = reason
diff --git a/hydrus/client/importing/ClientImportSubscriptions.py b/hydrus/client/importing/ClientImportSubscriptions.py
index 7703aef6..f1ddf259 100644
--- a/hydrus/client/importing/ClientImportSubscriptions.py
+++ b/hydrus/client/importing/ClientImportSubscriptions.py
@@ -112,6 +112,11 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
def _DelayWork( self, time_delta, reason ):
+ if len( reason ) > 0:
+
+ reason = reason.splitlines()[0]
+
+
self._no_work_until = HydrusData.GetNow() + time_delta
self._no_work_until_reason = reason
diff --git a/hydrus/client/importing/ClientImportWatchers.py b/hydrus/client/importing/ClientImportWatchers.py
index 1694f8d3..17e400e7 100644
--- a/hydrus/client/importing/ClientImportWatchers.py
+++ b/hydrus/client/importing/ClientImportWatchers.py
@@ -8,12 +8,12 @@ from hydrus.core import HydrusSerialisable
from hydrus.client import ClientConstants as CC
from hydrus.client import ClientData
+from hydrus.client.importing import ClientImportControl
from hydrus.client.importing import ClientImporting
from hydrus.client.importing import ClientImportFileSeeds
from hydrus.client.importing import ClientImportGallerySeeds
from hydrus.client.importing.options import ClientImportOptions
from hydrus.client.importing.options import FileImportOptions
-from hydrus.client.importing.options import PresentationImportOptions
from hydrus.client.importing.options import TagImportOptions
from hydrus.client.metadata import ClientTags
from hydrus.client.networking import ClientNetworkingJobs
@@ -32,7 +32,7 @@ class MultipleWatcherImport( HydrusSerialisable.SerialisableBase ):
self._lock = threading.Lock()
- self._page_key = 'initialising page key'
+ self._page_key = b'initialising page key'
self._watchers = HydrusSerialisable.SerialisableList()
@@ -565,7 +565,7 @@ class MultipleWatcherImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( self._page_key ):
+ if ClientImportControl.PageImporterShouldStopWorking( self._page_key ):
self._watchers_repeating_job.Cancel()
@@ -617,7 +617,7 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
HydrusSerialisable.SerialisableBase.__init__( self )
- self._page_key = 'initialising page key'
+ self._page_key = b'initialising page key'
self._publish_to_page = False
self._url = ''
@@ -650,7 +650,7 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
self._creation_time = HydrusData.GetNow()
self._file_velocity_status = ''
- self._file_status = ''
+ self._files_status = ''
self._watcher_status = ''
self._watcher_key = HydrusData.GenerateKey()
@@ -658,6 +658,8 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
self._have_started = False
self._lock = threading.Lock()
+ self._files_working_lock = threading.Lock()
+ self._checker_working_lock = threading.Lock()
self._last_pubbed_page_name = ''
@@ -701,12 +703,7 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- if len( text ) > 0:
-
- text = text.splitlines()[0]
-
-
- self._watcher_status = text
+ self._watcher_status = ClientImportControl.NeatenStatusText( text )
@@ -813,6 +810,11 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
def _DelayWork( self, time_delta, reason ):
+ if len( reason ) > 0:
+
+ reason = reason.splitlines()[0]
+
+
self._no_work_until = HydrusData.GetNow() + time_delta
self._no_work_until_reason = reason
@@ -1060,18 +1062,11 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
return
- did_substantial_work = False
-
def status_hook( text ):
with self._lock:
- if len( text ) > 0:
-
- text = text.splitlines()[0]
-
-
- self._file_status = text
+ self._files_status = ClientImportControl.NeatenStatusText( text )
@@ -1093,7 +1088,7 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- self._file_status = ''
+ self._files_status = ''
if did_substantial_work:
@@ -1304,29 +1299,12 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- gallery_work_to_do = self._gallery_seed_log.WorkToDo()
files_work_to_do = self._file_seed_cache.WorkToDo()
- gallery_go = gallery_work_to_do and not self._checking_paused
+ checker_go = HydrusData.TimeHasPassed( self._next_check_time ) and not self._checking_paused
files_go = files_work_to_do and not self._files_paused
- if self._watcher_status != '' or self._file_status != '':
-
- return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_WORKING, 'working' )
-
- elif gallery_go or files_go:
-
- return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_PENDING, 'pending' )
-
- elif self._checking_status == ClientImporting.CHECKER_STATUS_404:
-
- return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_DONE, '404' )
-
- elif self._checking_status == ClientImporting.CHECKER_STATUS_DEAD:
-
- return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_DONE, 'DEAD' )
-
- elif not HydrusData.TimeHasPassed( self._no_work_until ):
+ if not HydrusData.TimeHasPassed( self._no_work_until ):
if self._next_check_time is None:
@@ -1339,6 +1317,25 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_DEFERRED, text )
+ elif checker_go or files_go:
+
+ if self._checker_working_lock.locked() or self._files_working_lock.locked():
+
+ return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_WORKING, 'working' )
+
+ else:
+
+ return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_PENDING, 'pending' )
+
+
+ elif self._checking_status == ClientImporting.CHECKER_STATUS_404:
+
+ return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_DONE, '404' )
+
+ elif self._checking_status == ClientImporting.CHECKER_STATUS_DEAD:
+
+ return ( ClientImporting.DOWNLOADER_SIMPLE_STATUS_DONE, 'DEAD' )
+
else:
if self._checking_paused:
@@ -1364,17 +1361,7 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
with self._lock:
- file_status = self._file_status
-
- if self._checking_status == ClientImporting.CHECKER_STATUS_404:
-
- watcher_status = 'URL 404'
-
- elif self._checking_status == ClientImporting.CHECKER_STATUS_DEAD:
-
- watcher_status = 'URL DEAD'
-
- elif not HydrusData.TimeHasPassed( self._no_work_until ):
+ if not HydrusData.TimeHasPassed( self._no_work_until ):
if self._next_check_time is None:
@@ -1385,15 +1372,31 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
no_work_text = '{} - next check {}'.format( self._no_work_until_reason, ClientData.TimestampToPrettyTimeDelta( max( self._no_work_until, self._next_check_time ) ) )
- file_status = no_work_text
+ files_status = no_work_text
watcher_status = no_work_text
else:
- watcher_status = self._watcher_status
+ files_work_to_do = self._file_seed_cache.WorkToDo()
+
+ checker_go = HydrusData.TimeHasPassed( self._next_check_time ) and not self._checking_paused
+ files_go = files_work_to_do and not self._files_paused
+
+ if checker_go and not self._checker_working_lock.locked():
+
+ self._watcher_status = 'waiting for a work slot'
+
+
+ if files_go and not self._files_working_lock.locked():
+
+ self._files_status = 'waiting for a work slot'
+
+
+ files_status = ClientImportControl.GenerateLiveStatusText( self._files_status, self._files_paused, self._no_work_until, self._no_work_until_reason )
+ watcher_status = ClientImportControl.GenerateLiveStatusText( self._watcher_status, self._checking_paused, self._no_work_until, self._no_work_until_reason )
- return ( file_status, self._files_paused, self._file_velocity_status, self._next_check_time, watcher_status, self._subject, self._checking_status, self._check_now, self._checking_paused )
+ return ( files_status, self._files_paused, self._file_velocity_status, self._next_check_time, watcher_status, self._subject, self._checking_status, self._check_now, self._checking_paused )
@@ -1677,72 +1680,43 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
- def CanDoFileWork( self ):
+ def CheckCanDoFileWork( self ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( self._page_key ):
-
- self._files_repeating_job.Cancel()
-
- return
-
-
- files_paused = self._files_paused or HG.client_controller.new_options.GetBoolean( 'pause_all_file_queues' )
-
- if files_paused:
-
- return False
-
-
try:
- self._file_import_options.CheckReadyToImport()
+ ClientImportControl.CheckImporterCanDoWorkBecauseStopped( self._page_key )
- except Exception as e:
+ except HydrusExceptions.VetoException:
- self._file_status = str( e )
+ self._files_repeating_job.Cancel()
- HydrusData.ShowText( str( e ) )
-
- return False
+ raise
- work_to_do = self._file_seed_cache.WorkToDo()
+ ClientImportControl.CheckImporterCanDoFileWorkBecausePaused( self._files_paused, self._file_seed_cache, self._page_key )
- if not work_to_do:
+ try:
- return False
+ ClientImportControl.CheckImporterCanDoFileWorkBecausePausifyingProblem( self._file_import_options )
+
+ except HydrusExceptions.VetoException:
+
+ self._files_paused = True
+
+ raise
- return self.CanDoNetworkWork()
+ self.CheckCanDoNetworkWork()
- def CanDoNetworkWork( self ):
+ def CheckCanDoNetworkWork( self ):
with self._lock:
- no_delays = HydrusData.TimeHasPassed( self._no_work_until )
-
- if not no_delays:
-
- return False
-
-
- page_shown = not HG.client_controller.PageClosedButNotDestroyed( self._page_key )
-
- if not page_shown:
-
- return False
-
-
- network_engine_good = not HG.client_controller.network_engine.IsBusy()
-
- if not network_engine_good:
-
- return False
-
+ ClientImportControl.CheckCanDoNetworkWork( self._no_work_until, self._no_work_until_reason )
return True
@@ -1750,32 +1724,60 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
def REPEATINGWorkOnFiles( self ):
- while self.CanDoFileWork():
+ with self._files_working_lock:
- try:
+ while True:
- self._WorkOnFiles()
-
- HG.client_controller.WaitUntilViewFree()
-
- self._SerialisableChangeMade()
-
- except Exception as e:
-
- HydrusData.ShowException( e )
+ try:
+
+ try:
+
+ self.CheckCanDoFileWork()
+
+ except HydrusExceptions.VetoException as e:
+
+ with self._lock:
+
+ self._files_status = str( e )
+
+
+ break
+
+
+ self._WorkOnFiles()
+
+ HG.client_controller.WaitUntilViewFree()
+
+ self._SerialisableChangeMade()
+
+ except Exception as e:
+
+ with self._lock:
+
+ self._files_status = 'stopping work: {}'.format( str( e ) )
+
+
+ HydrusData.ShowException( e )
+
+ return
+
- def CanDoCheckerWork( self ):
+ def CheckCanDoCheckerWork( self ):
with self._lock:
- if ClientImporting.PageImporterShouldStopWorking( self._page_key ):
+ try:
+
+ ClientImportControl.CheckImporterCanDoWorkBecauseStopped( self._page_key )
+
+ except HydrusExceptions.VetoException:
self._checker_repeating_job.Cancel()
- return
+ raise
while self._gallery_seed_log.WorkToDo():
@@ -1788,46 +1790,78 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
self._gallery_seed_log.NotifyGallerySeedsUpdated( ( gallery_seed, ) )
- checking_paused = self._checking_paused or HG.client_controller.new_options.GetBoolean( 'pause_all_watcher_checkers' )
-
- if checking_paused:
+ if self._checking_paused:
- return False
+ raise HydrusExceptions.VetoException( 'paused' )
- able_to_check = self._checking_status == ClientImporting.CHECKER_STATUS_OK and self._HasURL()
-
- if not able_to_check:
+ if HG.client_controller.new_options.GetBoolean( 'pause_all_watcher_checkers' ):
- return False
+ raise HydrusExceptions.VetoException( 'all checkers are paused!' )
+
+
+ if not self._HasURL():
+
+ raise HydrusExceptions.VetoException( 'no url set yet!' )
+
+
+ if self._checking_status == ClientImporting.CHECKER_STATUS_404:
+
+ raise HydrusExceptions.VetoException( 'URL 404' )
+
+ elif self._checking_status == ClientImporting.CHECKER_STATUS_DEAD:
+
+ raise HydrusExceptions.VetoException( 'URL DEAD' )
check_due = HydrusData.TimeHasPassed( self._next_check_time )
if not check_due:
- return False
+ raise HydrusExceptions.VetoException( '' )
- return self.CanDoNetworkWork()
+ return self.CheckCanDoNetworkWork()
def REPEATINGWorkOnChecker( self ):
- if self.CanDoCheckerWork():
+ with self._checker_working_lock:
try:
+ try:
+
+ self.CheckCanDoCheckerWork()
+
+ except HydrusExceptions.VetoException as e:
+
+ with self._lock:
+
+ self._watcher_status = str( e )
+
+
+ return
+
+
self._CheckWatchableURL()
self._SerialisableChangeMade()
except Exception as e:
+ with self._lock:
+
+ self._watcher_status = 'stopping work: {}'.format( str( e ) )
+
+
HydrusData.ShowException( e )
+ return
+
+
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_WATCHER_IMPORT ] = WatcherImport
diff --git a/hydrus/client/importing/ClientImporting.py b/hydrus/client/importing/ClientImporting.py
index eacdfd40..1ec506f3 100644
--- a/hydrus/client/importing/ClientImporting.py
+++ b/hydrus/client/importing/ClientImporting.py
@@ -20,13 +20,13 @@ DOWNLOADER_SIMPLE_STATUS_PENDING = 2
DOWNLOADER_SIMPLE_STATUS_PAUSED = 3
DOWNLOADER_SIMPLE_STATUS_DEFERRED = 4
-downloader_enum_sort_lookup = {}
-
-downloader_enum_sort_lookup[ DOWNLOADER_SIMPLE_STATUS_DONE ] = 0
-downloader_enum_sort_lookup[ DOWNLOADER_SIMPLE_STATUS_WORKING ] = 1
-downloader_enum_sort_lookup[ DOWNLOADER_SIMPLE_STATUS_PENDING ] = 2
-downloader_enum_sort_lookup[ DOWNLOADER_SIMPLE_STATUS_DEFERRED ] = 3
-downloader_enum_sort_lookup[ DOWNLOADER_SIMPLE_STATUS_PAUSED ] = 4
+downloader_enum_sort_lookup = {
+ DOWNLOADER_SIMPLE_STATUS_DONE : 0,
+ DOWNLOADER_SIMPLE_STATUS_WORKING : 1,
+ DOWNLOADER_SIMPLE_STATUS_PENDING : 2,
+ DOWNLOADER_SIMPLE_STATUS_DEFERRED : 3,
+ DOWNLOADER_SIMPLE_STATUS_PAUSED : 4
+}
DID_SUBSTANTIAL_FILE_WORK_MINIMUM_SLEEP_TIME = 0.1
@@ -106,10 +106,6 @@ def GetRepeatingJobInitialDelay():
return 0.5 + ( random.random() * 0.5 )
-def PageImporterShouldStopWorking( page_key ):
-
- return HG.started_shutdown or not HG.client_controller.PageAlive( page_key )
-
def PublishPresentationHashes( publishing_label, hashes, publish_to_popup_button, publish_files_to_page ):
if publish_to_popup_button:
diff --git a/hydrus/client/networking/ClientLocalServerResources.py b/hydrus/client/networking/ClientLocalServerResources.py
index 837834e6..5791144c 100644
--- a/hydrus/client/networking/ClientLocalServerResources.py
+++ b/hydrus/client/networking/ClientLocalServerResources.py
@@ -309,9 +309,17 @@ def ParseClientAPIPOSTArgs( request ):
json_string = str( json_bytes, 'utf-8' )
- args = json.loads( json_string )
+ try:
+
+ args = json.loads( json_string )
+
+ except json.decoder.JSONDecodeError as e:
+
+ raise HydrusExceptions.BadRequestException( 'Sorry, did not understand the JSON you gave me: {}'.format( str( e ) ) )
+
parsed_request_args = ParseClientAPIPOSTByteArgs( args )
+
elif request_content_type_mime == HC.APPLICATION_CBOR:
diff --git a/hydrus/client/networking/ClientNetworkingJobs.py b/hydrus/client/networking/ClientNetworkingJobs.py
index 14a910e9..3c963dd9 100644
--- a/hydrus/client/networking/ClientNetworkingJobs.py
+++ b/hydrus/client/networking/ClientNetworkingJobs.py
@@ -19,6 +19,7 @@ from hydrus.core.networking import HydrusNetworking
from hydrus.client import ClientConstants as CC
from hydrus.client import ClientData
+from hydrus.client import ClientTime
from hydrus.client.networking import ClientNetworkingContexts
from hydrus.client.networking import ClientNetworkingFunctions
@@ -165,7 +166,8 @@ class NetworkJob( object ):
self._url = url
self._current_connection_attempt_number = 1
- self._max_connection_attempts_allowed = 5
+ self._current_request_attempt_number = 1
+ self._this_is_a_one_shot_request = False
self._we_tried_cloudflare_once = False
self._domain = ClientNetworkingFunctions.ConvertURLIntoDomain( self._url )
@@ -245,21 +247,33 @@ class NetworkJob( object ):
def _CanReattemptConnection( self ):
- return self._current_connection_attempt_number <= self._max_connection_attempts_allowed
+ if self._this_is_a_one_shot_request:
+
+ return False
+
+
+ max_connection_attempts_allowed = HG.client_controller.new_options.GetInteger( 'max_connection_attempts_allowed' )
+
+ return self._current_connection_attempt_number <= max_connection_attempts_allowed
def _CanReattemptRequest( self ):
+ if self._this_is_a_one_shot_request:
+
+ return False
+
+
if self._method == 'GET':
- max_attempts_allowed = 5
+ max_attempts_allowed = HG.client_controller.new_options.GetInteger( 'max_request_attempts_allowed_get' )
- elif self._method == 'POST':
+ else:
max_attempts_allowed = 1
- return self._current_connection_attempt_number <= max_attempts_allowed
+ return self._current_request_attempt_number <= max_attempts_allowed
def _GenerateModifiedDate( self, response: requests.Response ):
@@ -281,7 +295,12 @@ class NetworkJob( object ):
# the given struct is in GMT, so calendar.timegm is appropriate here
- self._response_last_modified = int( calendar.timegm( struct_time ) )
+ last_modified_time = int( calendar.timegm( struct_time ) )
+
+ if ClientTime.TimestampIsSensible( last_modified_time ):
+
+ self._response_last_modified = last_modified_time
+
except:
@@ -613,9 +632,9 @@ class NetworkJob( object ):
self.engine.bandwidth_manager.ReportDataUsed( self._network_contexts, num_bytes )
- def _ResetForAnotherConnectionAttempt( self ):
+ def _ResetForAnotherAttempt( self ):
- self._current_connection_attempt_number += 1
+ self._current_request_attempt_number += 1
self._content_type = None
self._response_mime = None
@@ -631,6 +650,14 @@ class NetworkJob( object ):
self._number_of_concurrent_empty_chunks = 0
+ def _ResetForAnotherConnectionAttempt( self ):
+
+ self._ResetForAnotherAttempt()
+
+ self._current_connection_attempt_number += 1
+ self._current_request_attempt_number = 1
+
+
def _SendRequestAndGetResponse( self ) -> requests.Response:
with self._lock:
@@ -974,7 +1001,9 @@ class NetworkJob( object ):
serverside_bandwidth_wait_time = HG.client_controller.new_options.GetInteger( 'serverside_bandwidth_wait_time' )
- self._serverside_bandwidth_wake_time = HydrusData.GetNow() + ( ( self._current_connection_attempt_number - 1 ) * serverside_bandwidth_wait_time )
+ problem_rating = ( self._current_connection_attempt_number + self._current_request_attempt_number ) - 1
+
+ self._serverside_bandwidth_wake_time = HydrusData.GetNow() + ( problem_rating * serverside_bandwidth_wait_time )
while not HydrusData.TimeHasPassed( self._serverside_bandwidth_wake_time ) and not self._IsCancelled():
@@ -1070,7 +1099,7 @@ class NetworkJob( object ):
with self._lock:
- if self._max_connection_attempts_allowed == 1:
+ if self._this_is_a_one_shot_request:
return True
@@ -1328,7 +1357,7 @@ class NetworkJob( object ):
def OnlyTryConnectionOnce( self ):
- self._max_connection_attempts_allowed = 1
+ self._this_is_a_one_shot_request = True
def OverrideBandwidth( self, delay = None ):
@@ -1589,7 +1618,7 @@ class NetworkJob( object ):
except HydrusExceptions.BandwidthException as e:
- self._ResetForAnotherConnectionAttempt()
+ self._ResetForAnotherAttempt()
if self._CanReattemptRequest():
@@ -1604,7 +1633,7 @@ class NetworkJob( object ):
except HydrusExceptions.ShouldReattemptNetworkException as e:
- self._ResetForAnotherConnectionAttempt()
+ self._ResetForAnotherAttempt()
if not self._CanReattemptRequest():
@@ -1615,7 +1644,7 @@ class NetworkJob( object ):
except requests.exceptions.ChunkedEncodingError:
- self._ResetForAnotherConnectionAttempt()
+ self._ResetForAnotherAttempt()
if not self._CanReattemptRequest():
@@ -1649,7 +1678,7 @@ class NetworkJob( object ):
except requests.exceptions.ReadTimeout:
- self._ResetForAnotherConnectionAttempt()
+ self._ResetForAnotherAttempt()
if not self._CanReattemptRequest():
diff --git a/hydrus/core/HydrusConstants.py b/hydrus/core/HydrusConstants.py
index fc34ca58..c387f3de 100644
--- a/hydrus/core/HydrusConstants.py
+++ b/hydrus/core/HydrusConstants.py
@@ -80,7 +80,7 @@ options = {}
# Misc
NETWORK_VERSION = 20
-SOFTWARE_VERSION = 488
+SOFTWARE_VERSION = 489
CLIENT_API_VERSION = 31
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )
diff --git a/hydrus/core/HydrusController.py b/hydrus/core/HydrusController.py
index 91a9df80..57a3ddaa 100644
--- a/hydrus/core/HydrusController.py
+++ b/hydrus/core/HydrusController.py
@@ -373,7 +373,7 @@ class HydrusController( object ):
def ClearCaches( self ) -> None:
- for cache in list(self._caches.values()): cache.Clear()
+ for cache in self._caches.values(): cache.Clear()
def CurrentlyIdle( self ) -> bool:
diff --git a/hydrus/core/HydrusData.py b/hydrus/core/HydrusData.py
index 29a8da36..b360d3af 100644
--- a/hydrus/core/HydrusData.py
+++ b/hydrus/core/HydrusData.py
@@ -1121,9 +1121,9 @@ def LastShutdownWasBad( db_path, instance ):
return False
-def MassUnion( lists ):
+def MassUnion( iterables ):
- return { item for item in itertools.chain.from_iterable( lists ) }
+ return { item for item in itertools.chain.from_iterable( iterables ) }
def MedianPop( population ):
diff --git a/hydrus/core/HydrusFileHandling.py b/hydrus/core/HydrusFileHandling.py
index 1b028686..30653184 100644
--- a/hydrus/core/HydrusFileHandling.py
+++ b/hydrus/core/HydrusFileHandling.py
@@ -327,6 +327,32 @@ def GetMime( path, ok_to_look_for_hydrus_updates = False ):
raise HydrusExceptions.ZeroSizeFileException( 'File is of zero length!' )
+ if ok_to_look_for_hydrus_updates and size < 64 * 1024 * 1024:
+
+ with open( path, 'rb' ) as f:
+
+ update_network_bytes = f.read()
+
+
+ try:
+
+ update = HydrusSerialisable.CreateFromNetworkBytes( update_network_bytes )
+
+ if isinstance( update, HydrusNetwork.ContentUpdate ):
+
+ return HC.APPLICATION_HYDRUS_UPDATE_CONTENT
+
+ elif isinstance( update, HydrusNetwork.DefinitionsUpdate ):
+
+ return HC.APPLICATION_HYDRUS_UPDATE_DEFINITIONS
+
+
+ except:
+
+ pass
+
+
+
with open( path, 'rb' ) as f:
bit_to_check = f.read( 256 )
@@ -360,6 +386,13 @@ def GetMime( path, ok_to_look_for_hydrus_updates = False ):
+ if HydrusText.LooksLikeHTML( bit_to_check ):
+
+ return HC.TEXT_HTML
+
+
+ # it is important this goes at the end, because ffmpeg has a billion false positives!
+ # for instance, it once thought some hydrus update files were mpegs
try:
mime = HydrusVideoHandling.GetMime( path )
@@ -379,37 +412,6 @@ def GetMime( path, ok_to_look_for_hydrus_updates = False ):
HydrusData.PrintException( e, do_wait = False )
- if ok_to_look_for_hydrus_updates:
-
- with open( path, 'rb' ) as f:
-
- update_network_bytes = f.read()
-
-
- try:
-
- update = HydrusSerialisable.CreateFromNetworkBytes( update_network_bytes )
-
- if isinstance( update, HydrusNetwork.ContentUpdate ):
-
- return HC.APPLICATION_HYDRUS_UPDATE_CONTENT
-
- elif isinstance( update, HydrusNetwork.DefinitionsUpdate ):
-
- return HC.APPLICATION_HYDRUS_UPDATE_DEFINITIONS
-
-
- except:
-
- pass
-
-
-
- if HydrusText.LooksLikeHTML( bit_to_check ):
-
- return HC.TEXT_HTML
-
-
return HC.APPLICATION_UNKNOWN
def GetThumbnailMime( path ):
diff --git a/hydrus/core/HydrusLogger.py b/hydrus/core/HydrusLogger.py
index 0b64ddf6..19b2cea3 100644
--- a/hydrus/core/HydrusLogger.py
+++ b/hydrus/core/HydrusLogger.py
@@ -50,7 +50,7 @@ class HydrusLogger( object ):
self._log_file.close()
- def _GetLogPath( self ) -> os.PathLike:
+ def _GetLogPath( self ) -> str:
current_time_struct = time.localtime()
diff --git a/hydrus/core/HydrusRatingArchive.py b/hydrus/core/HydrusRatingArchive.py
index 79b46c54..a3156114 100644
--- a/hydrus/core/HydrusRatingArchive.py
+++ b/hydrus/core/HydrusRatingArchive.py
@@ -44,7 +44,7 @@ HASH_TYPE_SHA512 = 3 # 64 bytes long
class HydrusRatingArchive( object ):
- def __init__( self, path : os.PathLike ) -> None:
+ def __init__( self, path : str ) -> None:
self._path = path
diff --git a/hydrus/core/HydrusVideoHandling.py b/hydrus/core/HydrusVideoHandling.py
index ca4c17ab..ec86539c 100644
--- a/hydrus/core/HydrusVideoHandling.py
+++ b/hydrus/core/HydrusVideoHandling.py
@@ -1,3 +1,5 @@
+import typing
+
import numpy
import numpy.typing
import os
@@ -984,7 +986,7 @@ def ParseFFMPEGVideoLine( lines, png_ok = False ) -> str:
return line
-def ParseFFMPEGVideoResolution( lines, png_ok = False ) -> tuple[int,int]:
+def ParseFFMPEGVideoResolution( lines, png_ok = False ) -> typing.Tuple[ int, int ]:
try: