RepoMirrors
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Version 435

This commit is contained in:
Hydrus Network Developer 2021-04-14 16:54:17 -05:00
parent 7048783704
commit a02c318cbb
25 changed files with 605 additions and 154 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
help/changelog.html
View File

@ -8,6 +8,35 @@
<div class="content"> <div class="content">
<h3 id="changelog"><a href="#changelog">changelog</a></h3> <h3 id="changelog"><a href="#changelog">changelog</a></h3>
<ul> <ul>
<li><h3 id="version_435"><a href="#version_435">version 435</a></h3></li>
<ul>
<li>misc:</li>
<li>a new macOS build that should run on Big Sur is now ready, it should be attached to this release. it is built on github automatically, and is thanks to hard work from Suika and ReAnzu. I am attaching my old release as well, just in case I messed up somewhere on my end. if you are a macOS user, please try the new App! it will not work on very old macOS like 10.12, but if this works out today for the majority of macOS users, I will be moving to just putting this new build out going forward. I'll add some polish like the readme.rtf and harmonise the filename etc.. too. I'd love to cut the filesize down, but this may not be possible (it is apparently some modern macOS thing where it bundles old and new versions of libraries in the same App so you basically get it twice)</li>
<li>the bottom-right corner of the regular media viewer canvas now also shows media zoom</li>
<li>the StringSorter object now has a simple 'reverse' sort type</li>
<li>the infamous multi-column list 'last column' width calculations are improved: first, dialogs with multi-column lists should no longer judder back and forth a single character's width as you expand the parent window. also, the last column saved size (which is used in dialog relaunch width initialisation) is now snapped to rounded 5-character intervals, which should mitigate various 'fuzzy' reasons for some dialogs to remember a larger or smaller size and grow or shrink one or more characters' width on the next launch</li>
<li>the _help->debug->gui actions_ menu has a new entry to reset all multi-column list saved widths back to default</li>
<li>the 'edit OR predicate' panel when you shift+double-click an OR predicate now expands horizontally and vertically with the window</li>
<li>the 'edit search predicates' list in the 'edit favourite search' panel now expands vertically with the window</li>
<li>the client now detects some invalid tag mapping states on tag upload--when a mapping is both current & pending or when it is both deleted and petitioned. these pair-states are mutually exclusive, normally impossible to get to, but one user who nonetheless ended up in this situation encountered an infinite uploading loop to a tag repository (since the tag was already current/deleted, the pending/petitioned status was not clearing correctly on upload commit). now, the upload will be abandoned and an info message put up with the fix</li>
<li>added a new maintenance routine to _database->check and repair_ that fixes logically inconsistent mappings. it has a popup dialog when it works and forces a pending count refresh and shows a summary afterwards</li>
<li>the routine that counts up total current or pending mappings on a service when the cached number has been reset is now massively faster (from a 30-60s down to less than a second in my dev tests). it now sums the tag autocomplete cache, rather than counting raw tables</li>
<li>fixed the BUGFIX option in 'connections' that allows you to disable ssl verification. this will also be extended at a later date to be domain-specific</li>
<li>.</li>
<li>new server stuff:</li>
<li>a new permission is added to hydrus service accounts--'manage options'. any account with 'manage account types' will get this by default on update</li>
<li>any account on a repository with 'manage options' permission will now see 'change update period' in the admin services menu! it launches a time delta control with the current update period and will send the new one up to the server. the client will resync account, options, and metadata immediately, and the server will generate any now-due updates immediately, so you should be able to watch changes occur in 'review services' and the server terminal live. other users will catch up to the new time when they next hit an update. various hardcoded check periods (like how often due updates are checked for and delay-buffered clientside and serverside) are shrunk significantly. the whole system should react to changes better</li>
<li>the minimum settable update time is now 10 minutes (the default value remains 100,000 seconds), but I recommend you try larger, say an hour minimum, at least to start. the network generally works more efficiently with higher numbers, and be warned, if you are adding 144 updates a day, there may be bloat problems after a year</li>
<li>let me know how this goes, whether you are running a server on a LAN or just a regular user running on one who gets a new update time!</li>
<li>the new 'full metadata resync' routine now triggers an immediate metadata update sync and wakes the daemon involved, so it should now happen as you watch</li>
<li>fixed the new pause/play buttons on review services to use neutral pause/play icons, not the downloader pause/play</li>
<li>brushed up metadata sync status string on review services</li>
<li>cleaned misc server and network code</li>
<li>cleaned up some old clientside service code</li>
<li>the client api now supports wildcard and namespace tags in the file search call</li>
<li>client api version is now 16</li>
<li>added https://ififfy.github.io/flipflip/#/ , a slideshow engine that now supports hydrus as a source, to the client api page</li>
</ul>
<li><h3 id="version_434"><a href="#version_434">version 434</a></h3></li> <li><h3 id="version_434"><a href="#version_434">version 434</a></h3></li>
<ul> <ul>
<li>network updates:</li> <li>network updates:</li>
@ -27,14 +56,14 @@
<li>fixed an issue with the new file sort asc/desc button where a transition from 'random' to another sort type using a favourite search would always reset the sort order to the top value</li> <li>fixed an issue with the new file sort asc/desc button where a transition from 'random' to another sort type using a favourite search would always reset the sort order to the top value</li>
<li>my asynchronous job object now has a default errback to catch errors more gracefully by default and with special handling in future. clicking an async button in a dialog will now show you the error there and then, rather than just the hidden error popup on the main window</li> <li>my asynchronous job object now has a default errback to catch errors more gracefully by default and with special handling in future. clicking an async button in a dialog will now show you the error there and then, rather than just the hidden error popup on the main window</li>
<li>added convenience links to the latest build on github to the help menu and html help</li> <li>added convenience links to the latest build on github to the help menu and html help</li>
<li>fixed another place in local file importing where a file that did not pass file import options checks would set 'skipped' status. in now sets 'ignored' like everything else</li> <li>fixed another place in local file importing where a file that did not pass file import options checks would set 'skipped' status. it now sets 'ignored' like everything else</li>
<li>fixed a bug when an 'undelete' call is sent to the media viewer when no media is set (usually during startup/shutdown)</li> <li>fixed a bug when an 'undelete' call is sent to the media viewer when no media is set (usually during startup/shutdown)</li>
<li>I disabled progress gauge 'pulsing' across the program. the way this was first implemented applied too often--I will bring it back to only apply when a job is both indeterminate and currently working</li> <li>I disabled progress gauge 'pulsing' across the program. the way this was first implemented applied too often--I will bring it back to only apply when a job is both indeterminate and currently working</li>
<li>my custom button class can now launch its own yes/no confirmation dialogs on click</li> <li>my custom button class can now launch its own yes/no confirmation dialogs on click</li>
<li>removed a subtag regen routine in the 425->426 update step that was bugging out due to bitrot--it now makes a popup message on boot asking for the routine to be run manually</li> <li>removed a subtag regen routine in the 425->426 update step that was bugging out due to bitrot--it now makes a popup message on boot asking for the routine to be run manually</li>
<li>fixed a typo bug in the 'subscription snapshot' debug command</li> <li>fixed a typo bug in the 'subscription snapshot' debug command</li>
<li>misc ancient python 2-to-3 code cleanup</li> <li>misc ancient python 2-to-3 code cleanup</li>
<li>updaned cloudscraper to 1.2.58</li> <li>updated cloudscraper to 1.2.58</li>
<li>.</li> <li>.</li>
<li>admin stuff:</li> <li>admin stuff:</li>
<li>.</li> <li>.</li>

8
help/client_api.html
View File

@ -18,7 +18,8 @@
<ul> <ul>
<li><a href="https://gitgud.io/prkc/hydrus-companion">https://gitgud.io/prkc/hydrus-companion</a> - Hydrus Companion, a Chrome/Firefox extension for hydrus that allows easy download queueing as you browse and advanced login support</li> <li><a href="https://gitgud.io/prkc/hydrus-companion">https://gitgud.io/prkc/hydrus-companion</a> - Hydrus Companion, a Chrome/Firefox extension for hydrus that allows easy download queueing as you browse and advanced login support</li>
<li><a href="https://github.com/floogulinc/hydrus-web">https://github.com/floogulinc/hydrus-web</a> - Hydrus Web, a web client for hydrus (allows phone browsing of hydrus)</li> <li><a href="https://github.com/floogulinc/hydrus-web">https://github.com/floogulinc/hydrus-web</a> - Hydrus Web, a web client for hydrus (allows phone browsing of hydrus)</li>
<li><a href="https://www.animebox.es/">https://www.animebox.es/</a> - Anime Boxes now supports adding your client as a Hydrus Server</li> <li><a href="https://www.animebox.es/">https://www.animebox.es/</a> - Anime Boxes, a booru browser, now supports adding your client as a Hydrus Server</li>
<li><a href="https://ififfy.github.io/flipflip/#/">https://ififfy.github.io/flipflip/#/</p> - FlipFlip, an advanced slideshow interface, now supports hydrus as a source</li>
<li><a href="https://gitgud.io/koto/hydrus-archive-delete">https://gitgud.io/koto/hydrus-archive-delete</a> - Archive/Delete filter in your web browser</li> <li><a href="https://gitgud.io/koto/hydrus-archive-delete">https://gitgud.io/koto/hydrus-archive-delete</a> - Archive/Delete filter in your web browser</li>
<li><a href="https://gitgud.io/koto/hydrus-dd">https://gitgud.io/koto/hydrus-dd</a> - DeepDanbooru neural network tagging for Hydrus</li> <li><a href="https://gitgud.io/koto/hydrus-dd">https://gitgud.io/koto/hydrus-dd</a> - DeepDanbooru neural network tagging for Hydrus</li>
<li><a href="https://gitgud.io/prkc/dolphin-hydrus-actions">https://gitgud.io/prkc/dolphin-hydrus-actions</a> - Adds Hydrus right-click context menu actions to Dolphin file manager.</li> <li><a href="https://gitgud.io/prkc/dolphin-hydrus-actions">https://gitgud.io/prkc/dolphin-hydrus-actions</a> - Adds Hydrus right-click context menu actions to Dolphin file manager.</li>
@ -42,7 +43,7 @@
</ul> </ul>
<p>There is now a simple 'session' system, where you can get a temporary key that gives the same access without having to include the permanent access key in every request. You can fetch a session key with the <a href="#session_key">/session_key</a> command and thereafter use it just as you would an access key, just with <i>Hydrus-Client-API-Session-Key</i> instead.</p> <p>There is now a simple 'session' system, where you can get a temporary key that gives the same access without having to include the permanent access key in every request. You can fetch a session key with the <a href="#session_key">/session_key</a> command and thereafter use it just as you would an access key, just with <i>Hydrus-Client-API-Session-Key</i> instead.</p>
<p>Session keys will expire if they are not used within 24 hours, or if the client is restarted, or if the underlying access key is deleted. An invalid/expired session key will give a <b>419</b> result with an appropriate error text.</p> <p>Session keys will expire if they are not used within 24 hours, or if the client is restarted, or if the underlying access key is deleted. An invalid/expired session key will give a <b>419</b> result with an appropriate error text.</p>
<p>Bear in mind the Client API is still under construction and is http-only for the moment--be careful about transmitting sensitive content outside of localhost. The access key will be unencrypted across any connection, and if it is included as a GET parameter, as simple and convenient as that is, it could be cached in all sorts of places.</p> <p>Bear in mind the Client API is still under construction. Setting up the Client API accessible across the internet requires technical experience to be convenient. HTTPS is available for encrypted comms, but the default certificate is self-signed (which basically means an eavesdropper can't see through it, but your ISP/government could if they decided to target you). If you have your own domain and SSL cert, you can replace them though (check the db directory for client.crt and client.key). Otherwise, be careful about transmitting sensitive content outside of your localhost/network.</p>
<h3 id="contents"><a href="#contents">Contents</a></h3> <h3 id="contents"><a href="#contents">Contents</a></h3>
<ul> <ul>
<li> <li>
@ -983,7 +984,8 @@
<li><p>/get_files/search_files?system_inbox=true&tags=%5B%22blue%20eyes%22%2C%20%22blonde%20hair%22%2C%20%22%5Cu043a%5Cu0438%5Cu043d%5Cu043e%22%5D</p></li> <li><p>/get_files/search_files?system_inbox=true&tags=%5B%22blue%20eyes%22%2C%20%22blonde%20hair%22%2C%20%22%5Cu043a%5Cu0438%5Cu043d%5Cu043e%22%5D</p></li>
</ul> </ul>
</li> </li>
<p>If the access key's permissions only permit search for certain tags, at least one whitelisted/non-blacklisted tag must be in the "tags" list or this will 403. Tags can be prepended with a hyphen to make a negated tag (e.g. "-green eyes"), but these will not be eligible for the permissions whitelist check.</p> <p>If the access key's permissions only permit search for certain tags, at least one whitelisted/non-blacklisted tag must be in the "tags" list or this will 403. Tags can be prepended with a hyphen to make a negated tag (e.g. "-green eyes"), but these will not be checked against the permissions whitelist.</p>
<p>Wildcards and namespace searches are now supported, so if you search for 'character:sam*' or 'series:*', this will be handled correctly clientside.</p>
<p>Response description: The full list of numerical file ids that match the search.</p> <p>Response description: The full list of numerical file ids that match the search.</p>
<li> <li>
<p>Example response:</p> <p>Example response:</p>

4
hydrus/client/ClientController.py
View File

@ -21,6 +21,7 @@ from hydrus.core import HydrusPaths
from hydrus.core import HydrusSerialisable from hydrus.core import HydrusSerialisable
from hydrus.core import HydrusThreading from hydrus.core import HydrusThreading
from hydrus.core import HydrusVideoHandling from hydrus.core import HydrusVideoHandling
from hydrus.core.networking import HydrusNetwork
from hydrus.core.networking import HydrusNetworking from hydrus.core.networking import HydrusNetworking
from hydrus.client import ClientAPI from hydrus.client import ClientAPI
@ -1184,9 +1185,10 @@ class Controller( HydrusController.HydrusController ):
job = self.CallRepeating( 5.0, 3600.0, self.SynchroniseAccounts ) job = self.CallRepeating( 5.0, 3600.0, self.SynchroniseAccounts )
job.ShouldDelayOnWakeup( True ) job.ShouldDelayOnWakeup( True )
job.WakeOnPubSub( 'notify_unknown_accounts' ) job.WakeOnPubSub( 'notify_unknown_accounts' )
job.WakeOnPubSub( 'notify_new_permissions' )
self._daemon_jobs[ 'synchronise_accounts' ] = job self._daemon_jobs[ 'synchronise_accounts' ] = job
job = self.CallRepeating( 5.0, 3600.0 * 4, self.SynchroniseRepositories ) job = self.CallRepeating( 5.0, HydrusNetwork.UPDATE_CHECKING_PERIOD, self.SynchroniseRepositories )
job.ShouldDelayOnWakeup( True ) job.ShouldDelayOnWakeup( True )
job.WakeOnPubSub( 'notify_restart_repo_sync' ) job.WakeOnPubSub( 'notify_restart_repo_sync' )
job.WakeOnPubSub( 'notify_new_permissions' ) job.WakeOnPubSub( 'notify_new_permissions' )

71
hydrus/client/ClientParsing.py
View File

@ -1865,6 +1865,7 @@ HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIAL
CONTENT_PARSER_SORT_TYPE_NONE = 0 CONTENT_PARSER_SORT_TYPE_NONE = 0
CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC = 1 CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC = 1
CONTENT_PARSER_SORT_TYPE_HUMAN_SORT = 2 CONTENT_PARSER_SORT_TYPE_HUMAN_SORT = 2
CONTENT_PARSER_SORT_TYPE_REVERSE = 3
class ContentParser( HydrusSerialisable.SerialisableBase ): class ContentParser( HydrusSerialisable.SerialisableBase ):
@ -3872,7 +3873,8 @@ HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIAL
sort_str_enum = { sort_str_enum = {
CONTENT_PARSER_SORT_TYPE_NONE : 'no sorting', CONTENT_PARSER_SORT_TYPE_NONE : 'no sorting',
CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC : 'strict lexicographic', CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC : 'strict lexicographic',
CONTENT_PARSER_SORT_TYPE_HUMAN_SORT : 'human sort' CONTENT_PARSER_SORT_TYPE_HUMAN_SORT : 'human sort',
CONTENT_PARSER_SORT_TYPE_REVERSE : 'reverse'
} }
class StringSorter( StringProcessingStep ): class StringSorter( StringProcessingStep ):
@ -3926,49 +3928,56 @@ class StringSorter( StringProcessingStep ):
texts = list( texts ) texts = list( texts )
data_convert = lambda d_s: d_s if self._sort_type == CONTENT_PARSER_SORT_TYPE_REVERSE:
invalid_data_convert_texts = []
if self._regex is not None:
re_job = re.compile( self._regex ) texts.reverse()
def d( d_s ): else:
data_convert = lambda d_s: d_s
invalid_data_convert_texts = []
if self._regex is not None:
m = re_job.search( d_s ) re_job = re.compile( self._regex )
if m is None: def d( d_s ):
return '' m = re_job.search( d_s )
else: if m is None:
return m.group() return ''
else:
return m.group()
data_convert = d
invalid_data_convert_texts = [ text for text in texts if data_convert( text ) == '' ]
texts = [ text for text in texts if data_convert( text ) != '' ]
data_convert = d sort_convert = lambda s: s
invalid_data_convert_texts = [ text for text in texts if data_convert( text ) == '' ] if self._sort_type == CONTENT_PARSER_SORT_TYPE_HUMAN_SORT:
texts = [ text for text in texts if data_convert( text ) != '' ]
sort_convert = HydrusData.HumanTextSortKey
key = lambda k_s: sort_convert( data_convert( k_s ) )
sort_convert = lambda s: s
if self._sort_type == CONTENT_PARSER_SORT_TYPE_HUMAN_SORT:
sort_convert = HydrusData.HumanTextSortKey reverse = not self._asc
texts.sort( key = key, reverse = reverse )
invalid_data_convert_texts.sort( key = sort_convert, reverse = reverse )
texts.extend( invalid_data_convert_texts )
key = lambda k_s: sort_convert( data_convert( k_s ) )
reverse = not self._asc
texts.sort( key = key, reverse = reverse )
invalid_data_convert_texts.sort( key = sort_convert, reverse = reverse )
texts.extend( invalid_data_convert_texts )
return texts return texts

14
hydrus/client/ClientServices.py
View File

@ -2124,8 +2124,11 @@ class ServiceRepository( ServiceRestricted ):
with self._lock: with self._lock:
self._next_account_sync = 1
self._do_a_full_metadata_resync = True self._do_a_full_metadata_resync = True
self._metadata.UpdateASAP()
self._SetDirty() self._SetDirty()
@ -2158,11 +2161,18 @@ class ServiceRepository( ServiceRestricted ):
def GetUpdateInfo( self ): def GetUpdatePeriod( self ):
with self._lock: with self._lock:
return self._metadata.GetUpdateInfo() if 'update_period' in self._service_options:
return self._service_options[ 'update_period' ]
else:
raise HydrusExceptions.DataMissing( 'This service does not seem to have an update period! Try refreshing your account!' )

157
hydrus/client/db/ClientDB.py
View File

@ -7617,6 +7617,92 @@ class DB( HydrusDB.HydrusDB ):
def _FixLogicallyInconsistentMappings( self, tag_service_key = None ):
job_key = ClientThreading.JobKey( cancellable = True )
total_fixed = 0
try:
job_key.SetVariable( 'popup_title', 'fixing logically inconsistent mappings' )
self._controller.pub( 'modal_message', job_key )
if tag_service_key is None:
tag_service_ids = self.modules_services.GetServiceIds( HC.REAL_TAG_SERVICES )
else:
tag_service_ids = ( self.modules_services.GetServiceId( tag_service_key ), )
for tag_service_id in tag_service_ids:
if job_key.IsCancelled():
break
message = 'fixing {}'.format( tag_service_id )
job_key.SetVariable( 'popup_text_1', message )
time.sleep( 0.01 )
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = ClientDBMappingsStorage.GenerateMappingsTableNames( tag_service_id )
#
both_current_and_pending_mappings = list(
HydrusData.BuildKeyToSetDict(
self._c.execute( 'SELECT tag_id, hash_id FROM {} CROSS JOIN {} USING ( tag_id, hash_id );'.format( pending_mappings_table_name, current_mappings_table_name ) )
).items()
)
total_fixed += sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in both_current_and_pending_mappings ) )
self._UpdateMappings( tag_service_id, pending_rescinded_mappings_ids = both_current_and_pending_mappings )
#
both_deleted_and_petitioned_mappings = list(
HydrusData.BuildKeyToSetDict(
self._c.execute( 'SELECT tag_id, hash_id FROM {} CROSS JOIN {} USING ( tag_id, hash_id );'.format( petitioned_mappings_table_name, deleted_mappings_table_name ) )
).items()
)
total_fixed += sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in both_deleted_and_petitioned_mappings ) )
self._UpdateMappings( tag_service_id, petitioned_rescinded_mappings_ids = both_deleted_and_petitioned_mappings )
finally:
if total_fixed == 0:
HydrusData.ShowText( 'No inconsistent mappings found!' )
else:
self._c.execute( 'DELETE FROM service_info where info_type IN ( ?, ? );', ( HC.SERVICE_INFO_NUM_PENDING_MAPPINGS, HC.SERVICE_INFO_NUM_PETITIONED_MAPPINGS ) )
self._controller.pub( 'notify_new_pending' )
HydrusData.ShowText( 'Found {} bad mappings! They _should_ be deleted, and your pending counts should be updated.'.format( HydrusData.ToHumanInt( total_fixed ) ) )
job_key.DeleteVariable( 'popup_text_2' )
job_key.SetVariable( 'popup_text_1', 'done!' )
job_key.Finish()
job_key.Delete( 5 )
def _GenerateDBJob( self, job_type, synchronous, action, *args, **kwargs ): def _GenerateDBJob( self, job_type, synchronous, action, *args, **kwargs ):
return JobDatabaseClient( job_type, synchronous, action, *args, **kwargs ) return JobDatabaseClient( job_type, synchronous, action, *args, **kwargs )
@ -11030,7 +11116,26 @@ class DB( HydrusDB.HydrusDB ):
pending_dict = HydrusData.BuildKeyToListDict( self._c.execute( 'SELECT tag_id, hash_id FROM ' + pending_mappings_table_name + ' ORDER BY tag_id LIMIT 100;' ) ) pending_dict = HydrusData.BuildKeyToListDict( self._c.execute( 'SELECT tag_id, hash_id FROM ' + pending_mappings_table_name + ' ORDER BY tag_id LIMIT 100;' ) )
for ( tag_id, hash_ids ) in list(pending_dict.items()): pending_mapping_ids = list( pending_dict.items() )
# dealing with a scary situation when (due to some bug) mappings are current and pending. they get uploaded, but the content update makes no changes, so we cycle infitely!
addable_pending_mapping_ids = self._FilterExistingUpdateMappings( service_id, pending_mapping_ids, HC.CONTENT_UPDATE_ADD )
pending_mapping_weight = sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in pending_mapping_ids ) )
addable_pending_mapping_weight = sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in addable_pending_mapping_ids ) )
if pending_mapping_weight != addable_pending_mapping_weight:
message = 'Hey, while going through the pending tags to upload, it seemed some were simultaneously already in the \'current\' state. This looks like a bug.'
message += os.linesep * 2
message += 'Please run _database->check and repair->fix logically inconsistent mappings_. If everything seems good after that and you do not get this message again, you should be all fixed. If not, you may need to regenerate your mappings storage cache under the \'database\' menu. If that does not work, hydev would like to know about it!'
HydrusData.ShowText( message )
raise HydrusExceptions.VetoException( 'Logically inconsistent mappings detected!' )
for ( tag_id, hash_ids ) in pending_mapping_ids:
tag = self.modules_tags_local_cache.GetTag( tag_id ) tag = self.modules_tags_local_cache.GetTag( tag_id )
hashes = self.modules_hashes_local_cache.GetHashes( hash_ids ) hashes = self.modules_hashes_local_cache.GetHashes( hash_ids )
@ -11042,7 +11147,30 @@ class DB( HydrusDB.HydrusDB ):
petitioned_dict = HydrusData.BuildKeyToListDict( [ ( ( tag_id, reason_id ), hash_id ) for ( tag_id, hash_id, reason_id ) in self._c.execute( 'SELECT tag_id, hash_id, reason_id FROM ' + petitioned_mappings_table_name + ' ORDER BY reason_id LIMIT 100;' ) ] ) petitioned_dict = HydrusData.BuildKeyToListDict( [ ( ( tag_id, reason_id ), hash_id ) for ( tag_id, hash_id, reason_id ) in self._c.execute( 'SELECT tag_id, hash_id, reason_id FROM ' + petitioned_mappings_table_name + ' ORDER BY reason_id LIMIT 100;' ) ] )
for ( ( tag_id, reason_id ), hash_ids ) in list(petitioned_dict.items()): petitioned_mapping_ids = list( petitioned_dict.items() )
# dealing with a scary situation when (due to some bug) mappings are deleted and petitioned. they get uploaded, but the content update makes no changes, so we cycle infitely!
deletable_and_petitioned_mappings = self._FilterExistingUpdateMappings(
service_id,
[ ( tag_id, hash_ids ) for ( ( tag_id, reason_id ), hash_ids ) in petitioned_mapping_ids ],
HC.CONTENT_UPDATE_DELETE
)
petitioned_mapping_weight = sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in petitioned_mapping_ids ) )
deletable_petitioned_mapping_weight = sum( ( len( hash_ids ) for ( tag_id, hash_ids ) in deletable_and_petitioned_mappings ) )
if petitioned_mapping_weight != deletable_petitioned_mapping_weight:
message = 'Hey, while going through the petitioned tags to upload, it seemed some were simultaneously already in the \'deleted\' state. This looks like a bug.'
message += os.linesep * 2
message += 'Please run _database->check and repair->fix logically inconsistent mappings_. If everything seems good after that and you do not get this message again, you should be all fixed. If not, you may need to regenerate your mappings storage cache under the \'database\' menu. If that does not work, hydev would like to know about it!'
HydrusData.ShowText( message )
raise HydrusExceptions.VetoException( 'Logically inconsistent mappings detected!' )
for ( ( tag_id, reason_id ), hash_ids ) in petitioned_mapping_ids:
tag = self.modules_tags_local_cache.GetTag( tag_id ) tag = self.modules_tags_local_cache.GetTag( tag_id )
hashes = self.modules_hashes_local_cache.GetHashes( hash_ids ) hashes = self.modules_hashes_local_cache.GetHashes( hash_ids )
@ -11127,7 +11255,7 @@ class DB( HydrusDB.HydrusDB ):
return media_result return media_result
petitioned = list(HydrusData.BuildKeyToListDict( self._c.execute( 'SELECT reason_id, hash_id FROM file_petitions WHERE service_id = ? ORDER BY reason_id LIMIT 100;', ( service_id, ) ) ).items()) petitioned = list( HydrusData.BuildKeyToListDict( self._c.execute( 'SELECT reason_id, hash_id FROM file_petitions WHERE service_id = ? ORDER BY reason_id LIMIT 100;', ( service_id, ) ) ).items() )
for ( reason_id, hash_ids ) in petitioned: for ( reason_id, hash_ids ) in petitioned:
@ -11640,9 +11768,27 @@ class DB( HydrusDB.HydrusDB ):
result = self._c.execute( 'SELECT COUNT( * ) FROM {};'.format( tags_table_name ) ).fetchone() result = self._c.execute( 'SELECT COUNT( * ) FROM {};'.format( tags_table_name ) ).fetchone()
elif info_type == HC.SERVICE_INFO_NUM_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + current_mappings_table_name + ';' ).fetchone() elif info_type in ( HC.SERVICE_INFO_NUM_MAPPINGS, HC.SERVICE_INFO_NUM_PENDING_MAPPINGS ):
ac_cache_table_name = self._CacheMappingsGetACCacheTableName( ClientTags.TAG_DISPLAY_STORAGE, self.modules_services.combined_file_service_id, service_id )
if info_type == HC.SERVICE_INFO_NUM_MAPPINGS:
column_name = 'current_count'
elif info_type == HC.SERVICE_INFO_NUM_PENDING_MAPPINGS:
column_name = 'pending_count'
result = self._c.execute( 'SELECT SUM( {} ) FROM {};'.format( column_name, ac_cache_table_name ) ).fetchone()
if result is None or result[0] is None:
result = ( 0, )
elif info_type == HC.SERVICE_INFO_NUM_DELETED_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + deleted_mappings_table_name + ';' ).fetchone() elif info_type == HC.SERVICE_INFO_NUM_DELETED_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + deleted_mappings_table_name + ';' ).fetchone()
elif info_type == HC.SERVICE_INFO_NUM_PENDING_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + pending_mappings_table_name + ';' ).fetchone()
elif info_type == HC.SERVICE_INFO_NUM_PETITIONED_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + petitioned_mappings_table_name + ';' ).fetchone() elif info_type == HC.SERVICE_INFO_NUM_PETITIONED_MAPPINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM ' + petitioned_mappings_table_name + ';' ).fetchone()
elif info_type == HC.SERVICE_INFO_NUM_PENDING_TAG_SIBLINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM tag_sibling_petitions WHERE service_id = ? AND status = ?;', ( service_id, HC.CONTENT_STATUS_PENDING ) ).fetchone() elif info_type == HC.SERVICE_INFO_NUM_PENDING_TAG_SIBLINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM tag_sibling_petitions WHERE service_id = ? AND status = ?;', ( service_id, HC.CONTENT_STATUS_PENDING ) ).fetchone()
elif info_type == HC.SERVICE_INFO_NUM_PETITIONED_TAG_SIBLINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM tag_sibling_petitions WHERE service_id = ? AND status = ?;', ( service_id, HC.CONTENT_STATUS_PETITIONED ) ).fetchone() elif info_type == HC.SERVICE_INFO_NUM_PETITIONED_TAG_SIBLINGS: result = self._c.execute( 'SELECT COUNT( * ) FROM tag_sibling_petitions WHERE service_id = ? AND status = ?;', ( service_id, HC.CONTENT_STATUS_PETITIONED ) ).fetchone()
@ -19298,6 +19444,7 @@ class DB( HydrusDB.HydrusDB ):
elif action == 'file_maintenance_add_jobs_hashes': self._FileMaintenanceAddJobsHashes( *args, **kwargs ) elif action == 'file_maintenance_add_jobs_hashes': self._FileMaintenanceAddJobsHashes( *args, **kwargs )
elif action == 'file_maintenance_cancel_jobs': self._FileMaintenanceCancelJobs( *args, **kwargs ) elif action == 'file_maintenance_cancel_jobs': self._FileMaintenanceCancelJobs( *args, **kwargs )
elif action == 'file_maintenance_clear_jobs': self._FileMaintenanceClearJobs( *args, **kwargs ) elif action == 'file_maintenance_clear_jobs': self._FileMaintenanceClearJobs( *args, **kwargs )
elif action == 'fix_logically_inconsistent_mappings': self._FixLogicallyInconsistentMappings( *args, **kwargs )
elif action == 'imageboard': self.modules_serialisable.SetYAMLDump( ClientDBSerialisable.YAML_DUMP_ID_IMAGEBOARD, *args, **kwargs ) elif action == 'imageboard': self.modules_serialisable.SetYAMLDump( ClientDBSerialisable.YAML_DUMP_ID_IMAGEBOARD, *args, **kwargs )
elif action == 'ideal_client_files_locations': self._SetIdealClientFilesLocations( *args, **kwargs ) elif action == 'ideal_client_files_locations': self._SetIdealClientFilesLocations( *args, **kwargs )
elif action == 'import_file': result = self._ImportFile( *args, **kwargs ) elif action == 'import_file': result = self._ImportFile( *args, **kwargs )

125
hydrus/client/gui/ClientGUI.py
View File

@ -60,6 +60,7 @@ from hydrus.client.gui import ClientGUIMPV
from hydrus.client.gui import ClientGUIPages from hydrus.client.gui import ClientGUIPages
from hydrus.client.gui import ClientGUIParsing from hydrus.client.gui import ClientGUIParsing
from hydrus.client.gui import ClientGUIPopupMessages from hydrus.client.gui import ClientGUIPopupMessages
from hydrus.client.gui import ClientGUIScrolledPanels
from hydrus.client.gui import ClientGUIScrolledPanelsEdit from hydrus.client.gui import ClientGUIScrolledPanelsEdit
from hydrus.client.gui import ClientGUIScrolledPanelsManagement from hydrus.client.gui import ClientGUIScrolledPanelsManagement
from hydrus.client.gui import ClientGUIScrolledPanelsReview from hydrus.client.gui import ClientGUIScrolledPanelsReview
@ -70,6 +71,7 @@ from hydrus.client.gui import ClientGUIStyle
from hydrus.client.gui import ClientGUISubscriptions from hydrus.client.gui import ClientGUISubscriptions
from hydrus.client.gui import ClientGUISystemTray from hydrus.client.gui import ClientGUISystemTray
from hydrus.client.gui import ClientGUITags from hydrus.client.gui import ClientGUITags
from hydrus.client.gui import ClientGUITime
from hydrus.client.gui import ClientGUITopLevelWindows from hydrus.client.gui import ClientGUITopLevelWindows
from hydrus.client.gui import ClientGUITopLevelWindowsPanels from hydrus.client.gui import ClientGUITopLevelWindowsPanels
from hydrus.client.gui import QtPorting as QP from hydrus.client.gui import QtPorting as QP
@ -1154,6 +1156,20 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ):
def _DebugResetColumnListManager( self ):
message = 'This will reset all saved column widths for all multi-column lists across the program. You may need to restart the client to see changes.'
result = ClientGUIDialogsQuick.GetYesNo( self, message )
if result != QW.QDialog.Accepted:
return
self._controller.column_list_manager.ResetToDefaults()
def _DebugShowGarbageDifferences( self ): def _DebugShowGarbageDifferences( self ):
count = collections.Counter() count = collections.Counter()
@ -1430,6 +1446,29 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ):
return -1 return -1
def _FixLogicallyInconsistentMappings( self ):
message = 'This will check for tags that are occupying mutually exclusive states--either current & pending or deleted & petitioned.'
message += os.linesep * 2
message += 'Please run this if you attempt to upload some tags and get a related error. You may need some follow-up regeneration work to correct autocomplete or \'num pending\' counts.'
result = ClientGUIDialogsQuick.GetYesNo( self, message, yes_label = 'do it--now choose which service', no_label = 'forget it' )
if result == QW.QDialog.Accepted:
try:
tag_service_key = GetTagServiceKeyForMaintenance( self )
except HydrusExceptions.CancelledException:
return
self._controller.Write( 'fix_logically_inconsistent_mappings', tag_service_key = tag_service_key )
def _FlipClipboardWatcher( self, option_name ): def _FlipClipboardWatcher( self, option_name ):
self._controller.new_options.FlipBoolean( option_name ) self._controller.new_options.FlipBoolean( option_name )
@ -2562,6 +2601,74 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ):
def _ManageServiceOptionsUpdatePeriod( self, service_key ):
service = self._controller.services_manager.GetService( service_key )
update_period = service.GetUpdatePeriod()
with ClientGUITopLevelWindowsPanels.DialogEdit( self, 'edit update period' ) as dlg:
panel = ClientGUIScrolledPanels.EditSingleCtrlPanel( dlg )
height_num_chars = 20
control = ClientGUITime.TimeDeltaCtrl( panel, min = HydrusNetwork.MIN_UPDATE_PERIOD, days = True, hours = True, minutes = True, seconds=True )
control.SetValue( update_period )
panel.SetControl( control )
dlg.SetPanel( panel )
if dlg.exec() == QW.QDialog.Accepted:
update_period = control.GetValue()
if update_period > HydrusNetwork.MAX_UPDATE_PERIOD:
QW.QMessageBox.information( self, 'Information', 'Sorry, the value you entered was too high. The max is {}.'.format( HydrusData.TimeDeltaToPrettyTimeDelta( HydrusNetwork.MAX_UPDATE_PERIOD ) ) )
return
job_key = ClientThreading.JobKey()
job_key.SetVariable( 'popup_title', 'setting update period' )
job_key.SetVariable( 'popup_text_1', 'uploading\u2026' )
self._controller.pub( 'message', job_key )
def work_callable():
service.Request( HC.POST, 'options_update_period', { 'update_period' : update_period } )
return 1
def publish_callable( gumpf ):
job_key.SetVariable( 'popup_text_1', 'done!' )
job_key.Finish()
service.DoAFullMetadataResync()
def errback_ui_cleanup_callable():
job_key.SetVariable( 'popup_text_1', 'error!' )
job_key.Finish()
job = ClientGUIAsync.AsyncQtJob( self, work_callable, publish_callable, errback_ui_cleanup_callable = errback_ui_cleanup_callable )
job.start()
def _ManageSubscriptions( self ): def _ManageSubscriptions( self ):
def qt_do_it( subscriptions, missing_query_log_container_names, surplus_query_log_container_names, original_pause_status ): def qt_do_it( subscriptions, missing_query_log_container_names, surplus_query_log_container_names, original_pause_status ):
@ -5087,6 +5194,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
ClientGUIMenus.AppendMenuItem( submenu, 'database integrity', 'Have the database examine all its records for internal consistency.', self._CheckDBIntegrity ) ClientGUIMenus.AppendMenuItem( submenu, 'database integrity', 'Have the database examine all its records for internal consistency.', self._CheckDBIntegrity )
ClientGUIMenus.AppendMenuItem( submenu, 'repopulate truncated mappings tables', 'Use the mappings cache to try to repair a previously damaged mappings file.', self._RepopulateMappingsTables ) ClientGUIMenus.AppendMenuItem( submenu, 'repopulate truncated mappings tables', 'Use the mappings cache to try to repair a previously damaged mappings file.', self._RepopulateMappingsTables )
ClientGUIMenus.AppendMenuItem( submenu, 'fix logically inconsistent mappings', 'Remove tags that are occupying two mutually exclusive states.', self._FixLogicallyInconsistentMappings )
ClientGUIMenus.AppendMenuItem( submenu, 'fix invalid tags', 'Scan the database for invalid tags.', self._RepairInvalidTags ) ClientGUIMenus.AppendMenuItem( submenu, 'fix invalid tags', 'Scan the database for invalid tags.', self._RepairInvalidTags )
ClientGUIMenus.AppendMenu( menu, submenu, 'check and repair' ) ClientGUIMenus.AppendMenu( menu, submenu, 'check and repair' )
@ -5265,7 +5373,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
ClientGUIMenus.AppendMenuItem( menu, 'review services', 'Look at the services your client connects to.', self._ReviewServices ) ClientGUIMenus.AppendMenuItem( menu, 'review services', 'Look at the services your client connects to.', self._ReviewServices )
ClientGUIMenus.AppendMenuItem( menu, 'manage services', 'Edit the services your client connects to.', self._ManageServices ) ClientGUIMenus.AppendMenuItem( menu, 'manage services', 'Edit the services your client connects to.', self._ManageServices )
repository_admin_permissions = [ ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_CREATE ), ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_MODERATE ), ( HC.CONTENT_TYPE_ACCOUNT_TYPES, HC.PERMISSION_ACTION_MODERATE ) ] repository_admin_permissions = [ ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_CREATE ), ( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_MODERATE ), ( HC.CONTENT_TYPE_ACCOUNT_TYPES, HC.PERMISSION_ACTION_MODERATE ), ( HC.CONTENT_TYPE_OPTIONS, HC.PERMISSION_ACTION_MODERATE ) ]
repositories = self._controller.services_manager.GetServices( HC.REPOSITORIES ) repositories = self._controller.services_manager.GetServices( HC.REPOSITORIES )
admin_repositories = [ service for service in repositories if True in ( service.HasPermission( content_type, action ) for ( content_type, action ) in repository_admin_permissions ) ] admin_repositories = [ service for service in repositories if True in ( service.HasPermission( content_type, action ) for ( content_type, action ) in repository_admin_permissions ) ]
@ -5285,10 +5393,13 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
service_key = service.GetServiceKey() service_key = service.GetServiceKey()
service_type = service.GetServiceType()
can_create_accounts = service.HasPermission( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_CREATE ) can_create_accounts = service.HasPermission( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_CREATE )
can_overrule_accounts = service.HasPermission( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_MODERATE ) can_overrule_accounts = service.HasPermission( HC.CONTENT_TYPE_ACCOUNTS, HC.PERMISSION_ACTION_MODERATE )
can_overrule_account_types = service.HasPermission( HC.CONTENT_TYPE_ACCOUNT_TYPES, HC.PERMISSION_ACTION_MODERATE ) can_overrule_account_types = service.HasPermission( HC.CONTENT_TYPE_ACCOUNT_TYPES, HC.PERMISSION_ACTION_MODERATE )
can_overrule_services = service.HasPermission( HC.CONTENT_TYPE_SERVICES, HC.PERMISSION_ACTION_MODERATE ) can_overrule_services = service.HasPermission( HC.CONTENT_TYPE_SERVICES, HC.PERMISSION_ACTION_MODERATE )
can_overrule_options = service.HasPermission( HC.CONTENT_TYPE_OPTIONS, HC.PERMISSION_ACTION_MODERATE )
if can_overrule_accounts: if can_overrule_accounts:
@ -5296,7 +5407,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
ClientGUIMenus.AppendMenuItem( submenu, 'modify an account', 'Modify a specific account\'s type and expiration.', self._ModifyAccount, service_key ) ClientGUIMenus.AppendMenuItem( submenu, 'modify an account', 'Modify a specific account\'s type and expiration.', self._ModifyAccount, service_key )
if can_overrule_accounts and service.GetServiceType() == HC.FILE_REPOSITORY: if can_overrule_accounts and service_type == HC.FILE_REPOSITORY:
ClientGUIMenus.AppendMenuItem( submenu, 'get an uploader\'s ip address', 'Fetch the ip address that uploaded a specific file, if the service knows it.', self._FetchIP, service_key ) ClientGUIMenus.AppendMenuItem( submenu, 'get an uploader\'s ip address', 'Fetch the ip address that uploaded a specific file, if the service knows it.', self._FetchIP, service_key )
@ -5315,7 +5426,14 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
ClientGUIMenus.AppendMenuItem( submenu, 'manage account types', 'Add, edit and delete account types for this service.', self._STARTManageAccountTypes, service_key ) ClientGUIMenus.AppendMenuItem( submenu, 'manage account types', 'Add, edit and delete account types for this service.', self._STARTManageAccountTypes, service_key )
if can_overrule_services and service.GetServiceType() == HC.SERVER_ADMIN: if can_overrule_options and service_type in HC.REPOSITORIES:
ClientGUIMenus.AppendSeparator( submenu )
ClientGUIMenus.AppendMenuItem( submenu, 'change update period', 'Change the update period for this service.', self._ManageServiceOptionsUpdatePeriod, service_key )
if can_overrule_services and service_type == HC.SERVER_ADMIN:
ClientGUIMenus.AppendSeparator( submenu ) ClientGUIMenus.AppendSeparator( submenu )
@ -5493,6 +5611,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
ClientGUIMenus.AppendMenuItem( gui_actions, 'refresh pages menu in five seconds', 'Delayed refresh the pages menu, giving you time to minimise or otherwise alter the client before it arrives.', self._controller.CallLater, 5, self._menu_updater_pages.update ) ClientGUIMenus.AppendMenuItem( gui_actions, 'refresh pages menu in five seconds', 'Delayed refresh the pages menu, giving you time to minimise or otherwise alter the client before it arrives.', self._controller.CallLater, 5, self._menu_updater_pages.update )
ClientGUIMenus.AppendMenuItem( gui_actions, 'publish some sub files in five seconds', 'Publish some files like a subscription would.', self._controller.CallLater, 5, lambda: HG.client_controller.pub( 'imported_files_to_page', [ HydrusData.GenerateKey() for i in range( 5 ) ], 'example sub files' ) ) ClientGUIMenus.AppendMenuItem( gui_actions, 'publish some sub files in five seconds', 'Publish some files like a subscription would.', self._controller.CallLater, 5, lambda: HG.client_controller.pub( 'imported_files_to_page', [ HydrusData.GenerateKey() for i in range( 5 ) ], 'example sub files' ) )
ClientGUIMenus.AppendMenuItem( gui_actions, 'make a parentless text ctrl dialog', 'Make a parentless text control in a dialog to test some character event catching.', self._DebugMakeParentlessTextCtrl ) ClientGUIMenus.AppendMenuItem( gui_actions, 'make a parentless text ctrl dialog', 'Make a parentless text control in a dialog to test some character event catching.', self._DebugMakeParentlessTextCtrl )
ClientGUIMenus.AppendMenuItem( gui_actions, 'reset multi-column list settings to default', 'Reset all multi-column list widths and other display settings to default.', self._DebugResetColumnListManager )
ClientGUIMenus.AppendMenuItem( gui_actions, 'force a main gui layout now', 'Tell the gui to relayout--useful to test some gui bootup layout issues.', self.adjustSize ) ClientGUIMenus.AppendMenuItem( gui_actions, 'force a main gui layout now', 'Tell the gui to relayout--useful to test some gui bootup layout issues.', self.adjustSize )
ClientGUIMenus.AppendMenuItem( gui_actions, 'save \'last session\' gui session', 'Make an immediate save of the \'last session\' gui session. Mostly for testing crashes, where last session is not saved correctly.', self.ProposeSaveGUISession, 'last session' ) ClientGUIMenus.AppendMenuItem( gui_actions, 'save \'last session\' gui session', 'Make an immediate save of the \'last session\' gui session. Mostly for testing crashes, where last session is not saved correctly.', self.ProposeSaveGUISession, 'last session' )

10
hydrus/client/gui/ClientGUICanvas.py
View File

@ -2183,15 +2183,19 @@ class CanvasWithDetails( Canvas ):
# bottom-right index # bottom-right index
bottom_right_string = ClientData.ConvertZoomToPercentage( self._current_zoom )
index_string = self._GetIndexString() index_string = self._GetIndexString()
if len( index_string ) > 0: if len( index_string ) > 0:
( text_size, index_string ) = ClientGUIFunctions.GetTextSizeFromPainter( painter, index_string ) bottom_right_string = '{} - {}'.format( bottom_right_string, index_string )
ClientGUIFunctions.DrawText( painter, my_width - text_size.width() - 3, my_height - text_size.height() - 3, index_string )
( text_size, bottom_right_string ) = ClientGUIFunctions.GetTextSizeFromPainter( painter, bottom_right_string )
ClientGUIFunctions.DrawText( painter, my_width - text_size.width() - 3, my_height - text_size.height() - 3, bottom_right_string )
def _GetInfoString( self ): def _GetInfoString( self ):

11
hydrus/client/gui/ClientGUIFunctions.py
View File

@ -36,12 +36,19 @@ def ColourIsGreyish( colour: QG.QColor ):
return it_is_greyish return it_is_greyish
def ConvertPixelsToTextWidth( window, pixels ): def ConvertPixelsToTextWidth( window, pixels, round_down = False ):
twenty_chars_in_pixels = int( window.fontMetrics().boundingRect( 20 * 'x' ).width() * MAGIC_TEXT_PADDING ) twenty_chars_in_pixels = int( window.fontMetrics().boundingRect( 20 * 'x' ).width() * MAGIC_TEXT_PADDING )
one_char_in_pixels = twenty_chars_in_pixels / 20 one_char_in_pixels = twenty_chars_in_pixels / 20
return round( pixels / one_char_in_pixels ) if round_down:
return int( pixels // one_char_in_pixels )
else:
return round( pixels / one_char_in_pixels )
def ConvertTextToPixels( window, char_dimensions ): def ConvertTextToPixels( window, char_dimensions ):

1
hydrus/client/gui/ClientGUIStringPanels.py
View File

@ -1498,6 +1498,7 @@ class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT ], ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT ) self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT ], ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT )
self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC ], ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC ) self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC ], ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC )
self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_REVERSE ], ClientParsing.CONTENT_PARSER_SORT_TYPE_REVERSE )
tt = 'Human sort sorts numbers as you understand them. "image 2" comes before "image 10". Lexicographic compares each character in turn. "image 02" comes before "image 10", which comes before "image 2".' tt = 'Human sort sorts numbers as you understand them. "image 2" comes before "image 10". Lexicographic compares each character in turn. "image 02" comes before "image 10", which comes before "image 2".'

38
hydrus/client/gui/lists/ClientGUIListCtrl.py
View File

@ -236,6 +236,12 @@ class BetterListCtrl( QW.QTreeWidget ):
last_column_index = num_columns - 1 last_column_index = num_columns - 1
# ok, the big pain in the ass situation here is getting a precise last column size that is reproduced on next dialog launch
# ultimately, with fuzzy sizing, style padding, scrollbars appearing, and other weirdness, the more precisely we try to define it, the more we will get dialogs that grow/shrink by a pixel each time
# *therefore*, the actual solution here is to move to snapping with a decent snap distance. the user loses size setting precision, but we'll snap back to a decent size every time, compensating for fuzz
LAST_COLUMN_SNAP_DISTANCE_CHARS = 5
for visual_index in range( num_columns ): for visual_index in range( num_columns ):
logical_index = header.logicalIndex( visual_index ) logical_index = header.logicalIndex( visual_index )
@ -244,15 +250,22 @@ class BetterListCtrl( QW.QTreeWidget ):
width_pixels = header.sectionSize( logical_index ) width_pixels = header.sectionSize( logical_index )
shown = not header.isSectionHidden( logical_index ) shown = not header.isSectionHidden( logical_index )
# if the scrollbar is in place, then when we initialise, next time, we will want to include that extra space in our final column recommended size if visual_index == last_column_index:
# might need to update this to be 'last non-hidden section', rather than 'last 'visual' section'
if visual_index == last_column_index and self.verticalScrollBar().isVisible():
width_pixels += self.verticalScrollBar().width() if self.verticalScrollBar().isVisible():
width_pixels += max( 0, min( self.verticalScrollBar().width(), 20 ) )
width_chars = ClientGUIFunctions.ConvertPixelsToTextWidth( main_tlw, width_pixels ) width_chars = ClientGUIFunctions.ConvertPixelsToTextWidth( main_tlw, width_pixels )
if visual_index == last_column_index:
# here's the snap magic
width_chars = round( width_chars // LAST_COLUMN_SNAP_DISTANCE_CHARS ) * LAST_COLUMN_SNAP_DISTANCE_CHARS
columns.append( ( column_type, width_chars, shown ) ) columns.append( ( column_type, width_chars, shown ) )
@ -307,7 +320,11 @@ class BetterListCtrl( QW.QTreeWidget ):
for i in range( self.topLevelItemCount() ): for i in range( self.topLevelItemCount() ):
if self.topLevelItem( i ).isSelected(): indices.append( i ) if self.topLevelItem( i ).isSelected():
indices.append( i )
return indices return indices
@ -747,21 +764,22 @@ class BetterListCtrl( QW.QTreeWidget ):
# the issue is: when we first boot up, we want to give a 'hey, it would be nice' size of the last actual recorded final column # the issue is: when we first boot up, we want to give a 'hey, it would be nice' size of the last actual recorded final column
# HOWEVER, after that: we want to use the current size of the last column # HOWEVER, after that: we want to use the current size of the last column
# so, if it is the first couple of seconds, lmao. after that, oaml # so, if it is the first couple of seconds, lmao. after that, oaml
# I later updated this to use the columnWidth, rather than hickery dickery text-to-pixel-width, since it was juddering resize around text width phase
last_column_type = self._column_list_status.GetColumnTypes()[-1] last_column_type = self._column_list_status.GetColumnTypes()[-1]
if HydrusData.TimeHasPassed( self._creation_time + 2 ): if HydrusData.TimeHasPassed( self._creation_time + 2 ):
last_column_chars = self._column_list_status.GetColumnWidth( last_column_type ) width += self.columnWidth( self.columnCount() - 1 )
else: else:
last_column_chars = self._original_column_list_status.GetColumnWidth( last_column_type ) last_column_chars = self._original_column_list_status.GetColumnWidth( last_column_type )
main_tlw = HG.client_controller.GetMainTLW()
main_tlw = HG.client_controller.GetMainTLW()
width += ClientGUIFunctions.ConvertTextToPixelWidth( main_tlw, last_column_chars )
width += ClientGUIFunctions.ConvertTextToPixelWidth( main_tlw, last_column_chars )
# #

7
hydrus/client/gui/lists/ClientGUIListManager.py
View File

@ -46,6 +46,13 @@ class ColumnListManager( HydrusSerialisable.SerialisableBase ):
return self._column_list_types_to_statuses[ column_list_type ] return self._column_list_types_to_statuses[ column_list_type ]
def ResetToDefaults( self ):
self._column_list_types_to_statuses = HydrusSerialisable.SerialisableDictionary()
self._dirty = True
def SaveStatus( self, column_list_status: ClientGUIListStatus.ColumnListStatus ): def SaveStatus( self, column_list_status: ClientGUIListStatus.ColumnListStatus ):
self._column_list_types_to_statuses[ column_list_status.GetColumnListType() ] = column_list_status self._column_list_types_to_statuses[ column_list_status.GetColumnListType() ] = column_list_status

2
hydrus/client/gui/search/ClientGUIACDropdown.py
View File

@ -1991,7 +1991,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
self._predicates_listbox = ListBoxTagsActiveSearchPredicates( self, self._page_key ) self._predicates_listbox = ListBoxTagsActiveSearchPredicates( self, self._page_key )
QP.AddToLayout( self._main_vbox, self._predicates_listbox, CC.FLAGS_EXPAND_PERPENDICULAR ) QP.AddToLayout( self._main_vbox, self._predicates_listbox, CC.FLAGS_EXPAND_BOTH_WAYS )
def _StartSearchResultsFetchJob( self, job_key ): def _StartSearchResultsFetchJob( self, job_key ):

2
hydrus/client/gui/search/ClientGUIPredicatesOR.py
View File

@ -37,7 +37,7 @@ class ORPredicateControl( QW.QWidget ):
vbox = QP.VBoxLayout() vbox = QP.VBoxLayout()
QP.AddToLayout( vbox, self._search_control, CC.FLAGS_CENTER_PERPENDICULAR ) QP.AddToLayout( vbox, self._search_control, CC.FLAGS_EXPAND_BOTH_WAYS )
self.setLayout( vbox ) self.setLayout( vbox )

11
hydrus/client/gui/search/ClientGUISearch.py
View File

@ -373,7 +373,16 @@ class EditPredicatesPanel( ClientGUIScrolledPanels.EditPanel ):
for panel in self._editable_pred_panels: for panel in self._editable_pred_panels:
QP.AddToLayout( vbox, panel, CC.FLAGS_EXPAND_PERPENDICULAR ) if isinstance( panel, ClientGUIPredicatesOR.ORPredicateControl ):
flags = CC.FLAGS_EXPAND_BOTH_WAYS
else:
flags = CC.FLAGS_EXPAND_PERPENDICULAR
QP.AddToLayout( vbox, panel, flags )
self.widget().setLayout( vbox ) self.widget().setLayout( vbox )

2
hydrus/client/gui/search/ClientGUISearchPanels.py
View File

@ -94,7 +94,7 @@ class EditFavouriteSearchPanel( ClientGUIScrolledPanels.EditPanel ):
vbox = QP.VBoxLayout() vbox = QP.VBoxLayout()
QP.AddToLayout( vbox, top_gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR ) QP.AddToLayout( vbox, top_gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
QP.AddToLayout( vbox, self._tag_autocomplete, CC.FLAGS_EXPAND_PERPENDICULAR ) QP.AddToLayout( vbox, self._tag_autocomplete, CC.FLAGS_EXPAND_BOTH_WAYS )
QP.AddToLayout( vbox, bottom_gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR ) QP.AddToLayout( vbox, bottom_gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
self.widget().setLayout( vbox ) self.widget().setLayout( vbox )

18
hydrus/client/gui/services/ClientGUIClientsideServices.py
View File

@ -2254,7 +2254,7 @@ class ReviewServiceRestrictedSubPanel( ClientGUICommon.StaticBox ):
self._rule_widgets = [] self._rule_widgets = []
self._network_sync_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().file_pause, self._PausePlayNetworkSync ) self._network_sync_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().pause, self._PausePlayNetworkSync )
self._network_sync_paused_button.setToolTip( 'pause/play account sync' ) self._network_sync_paused_button.setToolTip( 'pause/play account sync' )
self._refresh_account_button = ClientGUICommon.BetterButton( self, 'refresh account', self._RefreshAccount ) self._refresh_account_button = ClientGUICommon.BetterButton( self, 'refresh account', self._RefreshAccount )
@ -2350,11 +2350,11 @@ class ReviewServiceRestrictedSubPanel( ClientGUICommon.StaticBox ):
if self._service.IsPausedNetworkSync(): if self._service.IsPausedNetworkSync():
ClientGUIFunctions.SetBitmapButtonBitmap( self._network_sync_paused_button, CC.global_pixmaps().file_play ) ClientGUIFunctions.SetBitmapButtonBitmap( self._network_sync_paused_button, CC.global_pixmaps().play )
else: else:
ClientGUIFunctions.SetBitmapButtonBitmap( self._network_sync_paused_button, CC.global_pixmaps().file_pause ) ClientGUIFunctions.SetBitmapButtonBitmap( self._network_sync_paused_button, CC.global_pixmaps().pause )
# #
@ -2494,10 +2494,10 @@ class ReviewServiceRepositorySubPanel( ClientGUICommon.StaticBox ):
self._download_progress = ClientGUICommon.TextAndGauge( self ) self._download_progress = ClientGUICommon.TextAndGauge( self )
self._update_downloading_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().file_pause, self._PausePlayUpdateDownloading ) self._update_downloading_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().pause, self._PausePlayUpdateDownloading )
self._update_downloading_paused_button.setToolTip( 'pause/play update downloading' ) self._update_downloading_paused_button.setToolTip( 'pause/play update downloading' )
self._update_processing_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().file_pause, self._PausePlayUpdateProcessing ) self._update_processing_paused_button = ClientGUICommon.BetterBitmapButton( self, CC.global_pixmaps().pause, self._PausePlayUpdateProcessing )
self._update_processing_paused_button.setToolTip( 'pause/play update processing' ) self._update_processing_paused_button.setToolTip( 'pause/play update processing' )
self._processing_progress = ClientGUICommon.TextAndGauge( self ) self._processing_progress = ClientGUICommon.TextAndGauge( self )
@ -2702,22 +2702,22 @@ class ReviewServiceRepositorySubPanel( ClientGUICommon.StaticBox ):
if self._service.IsPausedUpdateDownloading(): if self._service.IsPausedUpdateDownloading():
ClientGUIFunctions.SetBitmapButtonBitmap( self._update_downloading_paused_button, CC.global_pixmaps().file_play ) ClientGUIFunctions.SetBitmapButtonBitmap( self._update_downloading_paused_button, CC.global_pixmaps().play )
else: else:
ClientGUIFunctions.SetBitmapButtonBitmap( self._update_downloading_paused_button, CC.global_pixmaps().file_pause ) ClientGUIFunctions.SetBitmapButtonBitmap( self._update_downloading_paused_button, CC.global_pixmaps().pause )
# #
if self._service.IsPausedUpdateProcessing(): if self._service.IsPausedUpdateProcessing():
ClientGUIFunctions.SetBitmapButtonBitmap( self._update_processing_paused_button, CC.global_pixmaps().file_play ) ClientGUIFunctions.SetBitmapButtonBitmap( self._update_processing_paused_button, CC.global_pixmaps().play )
else: else:
ClientGUIFunctions.SetBitmapButtonBitmap( self._update_processing_paused_button, CC.global_pixmaps().file_pause ) ClientGUIFunctions.SetBitmapButtonBitmap( self._update_processing_paused_button, CC.global_pixmaps().pause )
# #

27
hydrus/client/networking/ClientLocalServerResources.py
View File

@ -203,16 +203,33 @@ def ParseClientAPISearchPredicates( request ):
request.client_api_permissions.CheckCanSearchTags( tags ) request.client_api_permissions.CheckCanSearchTags( tags )
search_tags = [ ( True, tag ) for tag in tags ]
search_tags.extend( ( ( False, tag ) for tag in negated_tags ) )
predicates = [] predicates = []
for tag in negated_tags: for ( inclusive, tag ) in search_tags:
predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_TAG, value = tag, inclusive = False ) ) ( namespace, subtag ) = HydrusTags.SplitTag( tag )
if '*' in tag:
for tag in tags:
if subtag == '*':
tag = namespace
predicate_type = ClientSearch.PREDICATE_TYPE_NAMESPACE
else:
predicate_type = ClientSearch.PREDICATE_TYPE_WILDCARD
else:
predicate_type = ClientSearch.PREDICATE_TYPE_TAG
predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_TAG, value = tag ) ) predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_TAG, value = tag, inclusive = inclusive ) )
if system_inbox: if system_inbox:

5
hydrus/client/networking/ClientNetworkingSessions.py
View File

@ -296,6 +296,11 @@ class NetworkSessionManager( HydrusSerialisable.SerialisableBase ):
session.verify = False session.verify = False
if not HG.client_controller.new_options.GetBoolean( 'verify_regular_https' ):
session.verify = False
return session return session

7
hydrus/core/HydrusConstants.py
View File

@ -79,8 +79,8 @@ options = {}
# Misc # Misc
NETWORK_VERSION = 20 NETWORK_VERSION = 20
SOFTWARE_VERSION = 434 SOFTWARE_VERSION = 435
CLIENT_API_VERSION = 15 CLIENT_API_VERSION = 16
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 ) SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )
@ -322,6 +322,9 @@ permission_pair_string_lookup[ ( CONTENT_TYPE_ACCOUNTS, PERMISSION_ACTION_MODERA
permission_pair_string_lookup[ ( CONTENT_TYPE_ACCOUNT_TYPES, None ) ] = 'cannot change account types' permission_pair_string_lookup[ ( CONTENT_TYPE_ACCOUNT_TYPES, None ) ] = 'cannot change account types'
permission_pair_string_lookup[ ( CONTENT_TYPE_ACCOUNT_TYPES, PERMISSION_ACTION_MODERATE ) ] = 'can manage account types completely' permission_pair_string_lookup[ ( CONTENT_TYPE_ACCOUNT_TYPES, PERMISSION_ACTION_MODERATE ) ] = 'can manage account types completely'
permission_pair_string_lookup[ ( CONTENT_TYPE_OPTIONS, None ) ] = 'cannot change service options'
permission_pair_string_lookup[ ( CONTENT_TYPE_OPTIONS, PERMISSION_ACTION_MODERATE ) ] = 'can manage service options completely'
permission_pair_string_lookup[ ( CONTENT_TYPE_SERVICES, None ) ] = 'cannot change services' permission_pair_string_lookup[ ( CONTENT_TYPE_SERVICES, None ) ] = 'cannot change services'
permission_pair_string_lookup[ ( CONTENT_TYPE_SERVICES, PERMISSION_ACTION_MODERATE ) ] = 'can manage services completely' permission_pair_string_lookup[ ( CONTENT_TYPE_SERVICES, PERMISSION_ACTION_MODERATE ) ] = 'can manage services completely'

153
hydrus/core/networking/HydrusNetwork.py
View File

@ -10,6 +10,10 @@ from hydrus.core import HydrusSerialisable
from hydrus.core import HydrusTags from hydrus.core import HydrusTags
from hydrus.core.networking import HydrusNetworking from hydrus.core.networking import HydrusNetworking
UPDATE_CHECKING_PERIOD = 240
MIN_UPDATE_PERIOD = 600
MAX_UPDATE_PERIOD = 100000 * 100 # three months or so jej
def GenerateDefaultServiceDictionary( service_type ): def GenerateDefaultServiceDictionary( service_type ):
dictionary = HydrusSerialisable.SerialisableDictionary() dictionary = HydrusSerialisable.SerialisableDictionary()
@ -109,6 +113,7 @@ def GetPossiblePermissions( service_type ):
permissions.append( ( HC.CONTENT_TYPE_ACCOUNTS, [ None, HC.PERMISSION_ACTION_CREATE, HC.PERMISSION_ACTION_MODERATE ] ) ) permissions.append( ( HC.CONTENT_TYPE_ACCOUNTS, [ None, HC.PERMISSION_ACTION_CREATE, HC.PERMISSION_ACTION_MODERATE ] ) )
permissions.append( ( HC.CONTENT_TYPE_ACCOUNT_TYPES, [ None, HC.PERMISSION_ACTION_MODERATE ] ) ) permissions.append( ( HC.CONTENT_TYPE_ACCOUNT_TYPES, [ None, HC.PERMISSION_ACTION_MODERATE ] ) )
permissions.append( ( HC.CONTENT_TYPE_OPTIONS, [ None, HC.PERMISSION_ACTION_MODERATE ] ) )
if service_type == HC.FILE_REPOSITORY: if service_type == HC.FILE_REPOSITORY:
@ -770,7 +775,7 @@ class AccountType( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_ACCOUNT_TYPE SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_ACCOUNT_TYPE
SERIALISABLE_NAME = 'Account Type' SERIALISABLE_NAME = 'Account Type'
SERIALISABLE_VERSION = 1 SERIALISABLE_VERSION = 2
def __init__( def __init__(
self, self,
@ -852,6 +857,27 @@ class AccountType( HydrusSerialisable.SerialisableBase ):
self._auto_creation_history = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_auto_creation_history ) self._auto_creation_history = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_auto_creation_history )
def _UpdateSerialisableInfo( self, version, old_serialisable_info ):
if version == 1:
( serialisable_account_type_key, title, serialisable_permissions, serialisable_bandwidth_rules, auto_creation_velocity, serialisable_auto_creation_history ) = old_serialisable_info
permissions = dict( serialisable_permissions )
# admins can do options
if HC.CONTENT_TYPE_ACCOUNT_TYPES in permissions and permissions[ HC.CONTENT_TYPE_ACCOUNT_TYPES ] == HC.PERMISSION_ACTION_MODERATE:
permissions[ HC.CONTENT_TYPE_OPTIONS ] = HC.PERMISSION_ACTION_MODERATE
serialisable_permissions = list( permissions.items() )
new_serialisable_info = ( serialisable_account_type_key, title, serialisable_permissions, serialisable_bandwidth_rules, auto_creation_velocity, serialisable_auto_creation_history )
return ( 2, new_serialisable_info )
def BandwidthOK( self, bandwidth_tracker ): def BandwidthOK( self, bandwidth_tracker ):
return self._bandwidth_rules.CanStartRequest( bandwidth_tracker ) return self._bandwidth_rules.CanStartRequest( bandwidth_tracker )
@ -964,6 +990,7 @@ class AccountType( HydrusSerialisable.SerialisableBase ):
permissions[ HC.CONTENT_TYPE_ACCOUNTS ] = HC.PERMISSION_ACTION_MODERATE permissions[ HC.CONTENT_TYPE_ACCOUNTS ] = HC.PERMISSION_ACTION_MODERATE
permissions[ HC.CONTENT_TYPE_ACCOUNT_TYPES ] = HC.PERMISSION_ACTION_MODERATE permissions[ HC.CONTENT_TYPE_ACCOUNT_TYPES ] = HC.PERMISSION_ACTION_MODERATE
permissions[ HC.CONTENT_TYPE_OPTIONS ] = HC.PERMISSION_ACTION_MODERATE
if service_type in HC.REPOSITORIES: if service_type in HC.REPOSITORIES:
@ -1741,8 +1768,6 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_NAME = 'Metadata' SERIALISABLE_NAME = 'Metadata'
SERIALISABLE_VERSION = 1 SERIALISABLE_VERSION = 1
CLIENT_DELAY = 20 * 60
def __init__( self, metadata = None, next_update_due = None ): def __init__( self, metadata = None, next_update_due = None ):
if metadata is None: if metadata is None:
@ -1766,6 +1791,24 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
self._update_hashes = set() self._update_hashes = set()
self._biggest_end = self._CalculateBiggestEnd()
def _CalculateBiggestEnd( self ):
if len( self._metadata ) == 0:
return None
else:
biggest_index = max( self._metadata.keys() )
( update_hashes, begin, end ) = self._GetUpdate( biggest_index )
return end
def _GetNextUpdateDueTime( self, from_client = False ): def _GetNextUpdateDueTime( self, from_client = False ):
@ -1773,7 +1816,7 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
if from_client: if from_client:
delay = self.CLIENT_DELAY delay = UPDATE_CHECKING_PERIOD * 2
return self._next_update_due + delay return self._next_update_due + delay
@ -1818,6 +1861,8 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
self._update_hashes.update( update_hashes ) self._update_hashes.update( update_hashes )
self._biggest_end = self._CalculateBiggestEnd()
def AppendUpdate( self, update_hashes, begin, end, next_update_due ): def AppendUpdate( self, update_hashes, begin, end, next_update_due ):
@ -1831,6 +1876,23 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
self._next_update_due = next_update_due self._next_update_due = next_update_due
self._biggest_end = end
def CalculateNewNextUpdateDue( self, update_period ):
with self._lock:
if self._biggest_end is None:
self._next_update_due = 0
else:
self._next_update_due = self._biggest_end + update_period
def GetEarliestTimestampForTheseHashes( self, hashes ): def GetEarliestTimestampForTheseHashes( self, hashes ):
@ -1865,19 +1927,13 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
with self._lock: with self._lock:
keys = list( self._metadata.keys() ) if self._biggest_end is None:
if len( keys ) == 0:
return HydrusData.GetNow() return HydrusData.GetNow()
else: else:
largest_update_index = max( keys ) return self._biggest_end + 1
( update_hashes, begin, end ) = self._GetUpdate( largest_update_index )
return end + 1
@ -1890,20 +1946,24 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
return 'have not yet synced metadata' return 'have not yet synced metadata'
elif self._biggest_end is None:
return 'the metadata appears to be uninitialised'
else: else:
update_due = self._GetNextUpdateDueTime( from_client ) update_due = self._GetNextUpdateDueTime( from_client )
if HydrusData.TimeHasPassed( update_due ): if HydrusData.TimeHasPassed( update_due ):
s = 'imminently' s = 'next update imminent'
else: else:
s = HydrusData.TimestampToPrettyTimeDelta( update_due ) s = 'next update due {}'.format( HydrusData.TimestampToPrettyTimeDelta( update_due ) )
return 'next update due ' + s return 'metadata synced up to {}, {}'.format( HydrusData.TimestampToPrettyTimeDelta( self._biggest_end ), s )
@ -1982,45 +2042,6 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
def GetUpdateInfo( self, from_client = False ):
with self._lock:
num_update_hashes = sum( ( len( update_hashes ) for ( update_hashes, begin, end ) in list(self._metadata.values()) ) )
if len( self._metadata ) == 0:
status = 'have not yet synchronised'
else:
biggest_end = max( ( end for ( update_hashes, begin, end ) in list(self._metadata.values()) ) )
delay = 0
if from_client:
delay = self.CLIENT_DELAY
next_update_time = self._next_update_due + delay
if HydrusData.TimeHasPassed( next_update_time ):
s = 'imminently'
else:
s = HydrusData.TimestampToPrettyTimeDelta( next_update_time )
status = 'metadata synchronised up to ' + HydrusData.TimestampToPrettyTimeDelta( biggest_end ) + ', next update due ' + s
return ( num_update_hashes, status )
def HasDoneInitialSync( self ): def HasDoneInitialSync( self ):
with self._lock: with self._lock:
@ -2037,6 +2058,15 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
def UpdateASAP( self ):
with self._lock:
# not 0, that's reserved
self._next_update_due = 1
def UpdateDue( self, from_client = False ): def UpdateDue( self, from_client = False ):
with self._lock: with self._lock:
@ -2054,6 +2084,7 @@ class Metadata( HydrusSerialisable.SerialisableBase ):
self._metadata.update( metadata_slice._metadata ) self._metadata.update( metadata_slice._metadata )
self._next_update_due = metadata_slice._next_update_due self._next_update_due = metadata_slice._next_update_due
self._biggest_end = self._CalculateBiggestEnd()
@ -2477,6 +2508,20 @@ class ServerServiceRepository( ServerServiceRestricted ):
def SetUpdatePeriod( self, update_period: int ):
with self._lock:
self._service_options[ 'update_period' ] = update_period
self._metadata.CalculateNewNextUpdateDue( update_period )
self._SetDirty()
HG.server_controller.pub( 'notify_new_repo_sync' )
def Sync( self ): def Sync( self ):
with self._lock: with self._lock:

4
hydrus/server/ServerController.py
View File

@ -13,6 +13,7 @@ from hydrus.core import HydrusExceptions
from hydrus.core import HydrusGlobals as HG from hydrus.core import HydrusGlobals as HG
from hydrus.core import HydrusSessions from hydrus.core import HydrusSessions
from hydrus.core import HydrusThreading from hydrus.core import HydrusThreading
from hydrus.core.networking import HydrusNetwork
from hydrus.core.networking import HydrusNetworking from hydrus.core.networking import HydrusNetworking
from hydrus.server import ServerDB from hydrus.server import ServerDB
@ -257,7 +258,8 @@ class Controller( HydrusController.HydrusController ):
# #
job = self.CallRepeating( 5.0, 600.0, self.SyncRepositories ) job = self.CallRepeating( 5.0, HydrusNetwork.UPDATE_CHECKING_PERIOD, self.SyncRepositories )
job.WakeOnPubSub( 'notify_new_repo_sync' )
self._daemon_jobs[ 'sync_repositories' ] = job self._daemon_jobs[ 'sync_repositories' ] = job

17
hydrus/server/ServerDB.py
View File

@ -1574,12 +1574,6 @@ class DB( HydrusDB.HydrusDB ):
self._c.execute( 'INSERT OR IGNORE INTO ' + current_tag_parents_table_name + ' ( child_service_tag_id, parent_service_tag_id, account_id, parent_timestamp ) VALUES ( ?, ?, ?, ? );', ( child_service_tag_id, parent_service_tag_id, account_id, timestamp ) ) self._c.execute( 'INSERT OR IGNORE INTO ' + current_tag_parents_table_name + ' ( child_service_tag_id, parent_service_tag_id, account_id, parent_timestamp ) VALUES ( ?, ?, ?, ? );', ( child_service_tag_id, parent_service_tag_id, account_id, timestamp ) )
child_master_hash_ids = self._RepositoryGetCurrentMappingsMasterHashIds( service_id, child_service_tag_id )
overwrite_deleted = False
self._RepositoryAddMappings( service_id, account_id, parent_master_tag_id, child_master_hash_ids, overwrite_deleted, timestamp )
def _RepositoryAddTagSibling( self, service_id, account_id, bad_master_tag_id, good_master_tag_id, overwrite_deleted, timestamp ): def _RepositoryAddTagSibling( self, service_id, account_id, bad_master_tag_id, good_master_tag_id, overwrite_deleted, timestamp ):
@ -2070,17 +2064,6 @@ class DB( HydrusDB.HydrusDB ):
return count return count
def _RepositoryGetCurrentMappingsMasterHashIds( self, service_id, service_tag_id ):
( hash_id_map_table_name, tag_id_map_table_name ) = GenerateRepositoryMasterMapTableNames( service_id )
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateRepositoryMappingsTableNames( service_id )
master_hash_ids = [ master_hash_id for ( master_hash_id, ) in self._c.execute( 'SELECT master_hash_id FROM ' + hash_id_map_table_name + ' NATURAL JOIN ' + current_mappings_table_name + ' WHERE service_tag_id = ?;', ( service_tag_id, ) ) ]
return master_hash_ids
def _RepositoryGetFilesInfoFilesTableJoin( self, service_id, content_status ): def _RepositoryGetFilesInfoFilesTableJoin( self, service_id, content_status ):
( hash_id_map_table_name, tag_id_map_table_name ) = GenerateRepositoryMasterMapTableNames( service_id ) ( hash_id_map_table_name, tag_id_map_table_name ) = GenerateRepositoryMasterMapTableNames( service_id )

2
hydrus/server/networking/ServerServer.py
View File

@ -30,6 +30,8 @@ class HydrusServiceRestricted( HydrusServer.HydrusService ):
root.putChild( b'account_types', ServerServerResources.HydrusResourceRestrictedAccountTypes( self._service, HydrusServer.REMOTE_DOMAIN ) ) root.putChild( b'account_types', ServerServerResources.HydrusResourceRestrictedAccountTypes( self._service, HydrusServer.REMOTE_DOMAIN ) )
root.putChild( b'options_update_period', ServerServerResources.HydrusResourceRestrictedOptionsModifyUpdatePeriod( self._service, HydrusServer.REMOTE_DOMAIN ) )
root.putChild( b'registration_keys', ServerServerResources.HydrusResourceRestrictedRegistrationKeys( self._service, HydrusServer.REMOTE_DOMAIN ) ) root.putChild( b'registration_keys', ServerServerResources.HydrusResourceRestrictedRegistrationKeys( self._service, HydrusServer.REMOTE_DOMAIN ) )
return root return root

30
hydrus/server/networking/ServerServerResources.py
View File

@ -382,6 +382,36 @@ class HydrusResourceRestrictedOptions( HydrusResourceRestricted ):
return response_context return response_context
class HydrusResourceRestrictedOptionsModify( HydrusResourceRestricted ):
def _checkAccountPermissions( self, request: HydrusServerRequest.HydrusRequest ):
request.hydrus_account.CheckPermission( HC.CONTENT_TYPE_OPTIONS, HC.PERMISSION_ACTION_MODERATE )
class HydrusResourceRestrictedOptionsModifyUpdatePeriod( HydrusResourceRestrictedOptionsModify ):
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
update_period = request.parsed_request_args[ 'update_period' ]
if update_period < HydrusNetwork.MIN_UPDATE_PERIOD:
raise HydrusExceptions.BadRequestException( 'The update period was too low. It needs to be at least {}.'.format( HydrusData.TimeDeltaToPrettyTimeDelta( HydrusNetwork.MIN_UPDATE_PERIOD ) ) )
if update_period > HydrusNetwork.MAX_UPDATE_PERIOD:
raise HydrusExceptions.BadRequestException( 'The update period was too high. It needs to be lower than {}.'.format( HydrusData.TimeDeltaToPrettyTimeDelta( HydrusNetwork.MAX_UPDATE_PERIOD ) ) )
self._service.SetUpdatePeriod( update_period )
response_context = HydrusServerResources.ResponseContext( 200 )
return response_context
class HydrusResourceRestrictedAccountModify( HydrusResourceRestricted ): class HydrusResourceRestrictedAccountModify( HydrusResourceRestricted ):
def _checkAccountPermissions( self, request: HydrusServerRequest.HydrusRequest ): def _checkAccountPermissions( self, request: HydrusServerRequest.HydrusRequest ):