Version 515

closes #1324
This commit is contained in:
Hydrus Network Developer 2023-02-01 15:20:47 -06:00
parent 702900df96
commit 6af489a408
No known key found for this signature in database
GPG Key ID: 76249F053212133C
16 changed files with 492 additions and 380 deletions

View File

@ -7,6 +7,32 @@ title: Changelog
!!! note
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
## [Version 515](https://github.com/hydrusnetwork/hydrus/releases/tag/v515)
### related tags
* I worked on last week's related tags algorithm test, bringing it up to usable standard. the old buttons now use the new algorithm exclusively. all users now get 'related tags' showing in manage tags by default (if you don't like it, you can turn it off under _options->tag suggestions_)
* the new algorithm has new cancel tech and does a 'work for 600ms' kind of deal, like the old system, and the last-minute blocks from last week are gone--it will search as much as it has time for, including partial results. it also won't lag you out for thirty seconds (unless you tell it to in the options). it searches tags with low count first, so don't worry if it doesn't get to everything--'1girl' usually doesn't have a huge amount extra to offer once everything else has run
* it also uses 'hydev actually thought about this' statistical sampling tech to work massively faster on larger-count tags at the cost of some variance in rank and the odd false positive (considered sufficiently related when it actually shouldn't meet the threshold) nearer the bottom end of the tags result list
* rather than 'new 1' and 'new 2', there is now an on/off button for searching your local files or all known files on tag repositories. 'all known files' = great results, but very slow, which the tooltip explains
* there's also a new status label that will tell you when it is searching and how well the search went (e.g. '12/51 tags searched fully in 459ms')
* I also added the 'quick' search button back in, since we can now repeat searches for just selections of tags
* I fixed a couple typos in the algorthim that were messing some results
* in the manage tags dialog, if you have the suggested tag panels 'side-to-side', they now go in named boxes
* in the manage tags dialog, if you have suggested tag panels in a notebook, 'related tags' will only refresh its search on a media change event (including dialog initialisation) when it is the selected page. it won't lag you from the background!
* options->tag suggestions now lets you pick which notebook'd tag suggestions page you want to show by default. this defaults to 'related'
* I have more plans here. these related tags results are very cachable, so that's an obvious next step to speed up results, and when I have done some other long-term tag improvements elsewhere in the program, I'll be able to quickly filter out unhelpful sibling and parent suggestions. more immediately, I think we'll want some options for namespace weighting (e.g. 'series:' tags' suggestions could have higher rank than 'smile'), so we can tune things a bit
### misc
* the 'open externally' canvas widget, which shows any available thumbnail of the flash or psd or whatever, now sizes itself correctly and draws the thumbnail nicely if you set the new thumbnail supersampling option to >100%. if your thumbnail is the wrong size (and probably in a queue to be regenerated soon), I _think_ it'll still make the window too big/small, but it'll draw the thumbnail to fit
* if a tag content update comes in with an invalid tag (such as could happen with sidecars recently), the client now heals better. the bad tag is corrected live in more places, and this should be propagated to the UI. if you got a warning about 'you have invalid tags in view' recently but running the routine found no problems, please reboot, and I think you'll be fixed. I'm pretty sure the database wasn't being damaged at all here (it has cleaning safeguards, so it _shouldn't_ be possible to actually save bad tags)--it was just a thing to do with the UI not being told of the cleaned tag, and it shouldn't happen again. thank you for the reports! (issue #1324)
* export folders and the file maintenance dialog no longer apply the implicit system:limit (defaults to max 10k files) to their searches!
* old OR predicates that you load with saved searches and similar should now always have alphebetised components, and if you double-click them to remove them, they will now clear correctly (previously, they were doing something similar to the recent filetype problem, where instead of recognising themselves and deleting, they would instead duplicate a normalised (sorted) copy of themselves)
* thanks to a user, updated the recently note-and-ai-updated pixiv parser again to grab the canonical pixiv URL and translated tags, if present
* thanks to a user, updated the sankaku parser to grab some more tags
* the file location context and tag context buttons under tag autocompletes now put menu separators between each type of file/tag service in their menus. for basic users, this'll be a separator for every row, but for advanced users with multiple local domains, it will help categorise the list a bit
## [Version 514](https://github.com/hydrusnetwork/hydrus/releases/tag/v514)
### downloaders
@ -61,8 +87,7 @@ title: Changelog
* **`hide_service_keys_tags` is now default true. it will be removed in 4 weeks or so. same deal as with `service_names_to_statuses_to_...`--move to `tags`**
* **`system_inbox` and `system_archive` are removed from `/get_files/search_files`! just use 'system:inbox/archive' in the tags list**
* **the 'set_file_relationships' command from last week has been reworked to have a nicer Object parameter with a new name. please check the updated help!** normally I wouldn't change something so quick, but we are still in early prototype, so I'm ok shifting it (and the old method still works lmao, but I'll clear that code out in a few weeks, so please move over--the Object will be much nicer to expand in future, which I forgot about in v513)
### many Client API commands now support modern file domain objects, meaning you can search a UNION of file services and 'deleted-from' file services. affected commands are
* many Client API commands now support modern file domain objects, meaning you can search a UNION of file services and 'deleted-from' file services. affected commands are
* * /add_files/delete_files
* * /add_files/undelete_files
* * /add_tags/search_tags
@ -441,81 +466,3 @@ title: Changelog
* cleaned up some canvas zoom code
* fixed another 'duplicates' unit test that would on rare occasion fail due to a too-specific test
* removed a no-longer needed token declaration from the github build script that was raising a warning
## [Version 505](https://github.com/hydrusnetwork/hydrus/releases/tag/v505)
### exif update
* the client now has the ability to check your image files for basic human-readable metadata. sometimes this is timing data for a gif, often it is something like DPI, and for many of the recent ML-generated pngs, this is the original generating prompt. this is now viewable in the same way as EXIF, on the same panel. since this (and future expansions) are not EXIF _per se_, the overarching UI around here is broadly renamed 'embedded metadata'
* the client now scans for and remembers if files have EXIF or human-readable embedded metadata. two predicates, 'system:image has exif' and 'system:image has human-readable embedded metadata' let you search for them. the vast majority of images have some sort of human-readable embedded metadata, so 'system:no human-readable embedded metadata' may typically be the more useful predicate in the latter case
* the system predicate parser can handle these new system preds
* to keep the system predicate list tidy, the new system preds are wrapped with 'has icc profile' into a meta-system predicate 'system:embedded metadata', like how 'system:dimensions' works
* the media viewer now knows ahead of time if a media has embedded metadata. the button in the media viewer's top hover window that shows this is no longer a cog but a little text-on-window image, and it now only appears if the file has data to show. the tooltip previews whether this is EXIF, other data, or both
* this knowledge is obviously now generated on file imports going forward, and new file maintenance jobs can retroactively scan for it
* all your existing image files and gifs/apngs are scheduled for this work. they will catch up in the background over the coming weeks
* the duplicate filter shows if one or both files have exif or other human-readable data. I had written off adding new 'scores' to the dupe filter panel until a full overhaul, but this was a simple copy/paste of the icc profile statement, so I snuck it in. also, these statements now only appear if for one image it is true and the other is false--no more 'they both have icc profiles m8', which is not a helpful comparison statement
* added some unit tests for this new tech
* a future expansion here will be to record the specific keys and values into the database so you can search specifically over those values (e.g. 'EXIF ISO level > 400', or 'has "parameters" text value')
### misc
* the 'reverse page drop shift behaviour' checkbox in _options->gui pages_ is replaced with four checkboxes. two govern whether page drops should chase the drop, either normally or with shift held down, and two new ones govern whether hydrus should dynamically navigate tabs as you move a media or page drag and drop over the tab bar. set them how you like!
* a new EXPERIMENTAL checkbox just beneath these lets you change what the mouse wheel does to a row of page tabs--by default, the wheel will change tab selection, but if you often have an overloaded row (i.e. they overspill the bar width and you see the left/right arrows), you can set the wheel to _scroll/pan the bar_ instead
* the 'if file is missing, remove record' job is now split into two--one that leaves no deletion record (old behaviour), and one that does (new). this new job lets you do some 'yes and I want it to stay gone' tasks like if you are syncing an old database backup to a newer client_files structure
* thanks to user pointing out what was needed, turned on a beta 'darkmode detection' in Qt for Windows. if you launch the client in official Windows 'Apps darkmode' (under Windows settings->Colors), it should now start with your system darkmode colours. switching between light and dark mode while the client is running is pretty buggy (also my Explorer windows are buggy at this too jej), but this is a step forward. fingers crossed this feature matures and gets reliable multiplatform support in future (issue #756)
### fixes
* thanks to a user, the twitter downloader is fixed. seems like twitter (maybe due to Elon's new team?) changed one tiny name in the API we use. let's see if they change anything more significant in the coming weeks (issue #1268)
* thanks to a user the 'gelbooru 0.1.11 file page parser' stops getting borked 'Rating: ' tags, and I fixed its source time fetch too. I'm pretty sure these broke because of the multiline string processing change a couple months ago, sorry for the trouble!
* fixed a recent stupid typo that broke the media viewer's do an edge pan' action (issue #1266)
* fixed an issue with the furry.booru.org url classes, which were normalising URLs to http rather than https for some accidental reason
* I finally figured out the weird bug where the colour picker dialog would sometimes treat mouse moves as mouse drags over the colour-selection gradient box. this is due to a bug in Qt6 where if you have a stylesheet with a certain hover value set, the colour picker goes bananas. I tried many things to fix this and finally settled on a sledgehammer: if you have the offending value in your stylesheet, it now does some stuff that takes a second or two of lag to launch the colour picker and a second or two of lag to exit it. sorry, but that fixes it! if you want to skip the lag in the options dialog, set your stylesheet to 'default' for the duration (issue #1260)
* fixed an issue where the new sidecar importer system was not correctly cleaning tags (removing extra whitespace, lowercasing) before committing them to the database! if you got hit with this, a simple restart should fix the incorrect labels (it wasn't _actually_ writing bad tags to the database), but if a restart does not fix it, please run _database->check and repair->fix invalid tags_ (issue #1264)
* fixed an issue opening the new metadata sidecar edit UI when you had removed and replaced the original 'my tags' service
* think I fixed a bug in the duplicate filter where if a file in the current pair is deleted (and removed from view), the index/pair tracking would desynchronise and cause an error if you attempted to rewind to the first pair
* I fixed the reported 'committable decisions' count for duplicate filters set to do no duplicate content merge at all
### build version woes
* all the builds now run on python 3.9 (Linux and Windows were 3.8 previously). any users on systems too old to run 3.9 are encouraged to run from source instead
* the linux build is rolled back to the older version of python-mpv. thanks to the users who helped me test this, and the specific user who let me know about the different version incompatibilities going on. basically we can't move to the new mpv on the Linux build for a little while, so the official release is rolling back to safe and stable. if you are on a newer Linux flavour, like 22.04, I recommend you pursue running from source, which is now easy on Linux
* I am considering, in let's say two or three months, no longer supporting the Linux build. we'll see how well the running from source easy-setup scripts work out, but if they aren't a hassle, that really is the proper way to do things on Linux, and it'll solve many crashes and mpv issues
### running from source is now simple and easy for everyone
* transcribed the setup .bat files in the base directory to .sh for linux users and .command for macOS users! the 'running from source' help is updated too. all users are now welcome to try it out!
* folded the 'setup_venv_qt5.bat' script into the main 'setup_venv.bat' script as a user choice for 'advanced' setup, and expanded it with prompts for qt5, mpv, and opencv
* the setup files now say your python version and guide you through all choices
* as Windows 8.1 users have reported problems with Qt6, the help and script recommendations on Qt5 are now <=8.1, not just 7. but it is easy to switch now, so if you want to play around, let me know what you discover
### boring running from source and help gubbins
* took the 'update' option out of the 'setup-venv.bat' script. this process was not doing what I thought it would and was not particularly useful. the script now always reinstalls after user hits Enter to continue, which is very reliable, gets newer versions of libraries when available, and almost always takes less than a minute
* updated the github readme and website index to point obviously and directly at the getting started guide
* took out some of the bloviating from the initial introduction page
* updated the running from source help to talk about the new advanced setup and added a couple extra warnings
* updated the running from source help to talk about Linux and macOS
* if qtpy is missing at the very start of the program, a new error catch asks the user if they installed and activated their venv correctly (should also catch people who run client.py right off the bat without reading the docs)
* deleted the old user-written help document about which packages to use with which Linux flavours, as the author says it is now out of date and modern pip as used by the scripts navigates it better nowadays
* the setup_venv.bat now checks and informs the user if they do not have python installed
* cleaned up the flow control of the batch files. more conditionals, fewer gotos
* to keep the base install dir clean, moved the 'advanced' setup script's cut-up requirements.txts to a new folder under static/requirements. if you are manually setting up a venv and need unusual libraries, check them out for known good specific versions, otherwise you are set with the basic requirements.txt
* to keep the install dir clean, moved the obscure 'build' requirements.txts to a new folder under static/requirements. these are mostly just notes for me when setting up a new test dev environment
### cleanup and other boring stuff
* as recommended by the pyopenssl page, I moved the server self-signed cert generation routine to 'cryptography' (which I'm pretty sure pyopenssl was just wrapping anyway). cryptography is added to the requirements.txt, but you should already have it. pyopenssl is still used by twisted, so it stays in the requirements.txts. both of these libraries remain optional and are only used by people hosting https services
* if you load up a favourite search, the focus no longer goes to the autocomplete text box right after. hydev liked most of the focus propagation changes here but found this one incredibly annoying
* when you are in profile mode and doing repository processing, the current speed is now printed regularly to the profile log to help see how fast the profiled jobs are at each step
* simplified some duplicate filter code
* the 'add tags/urls with the import' window now also shows 'cleaned' tags in the preview column for sidecar routers that go to tags
* added some extra help text and tooltips to the new sidecar exporter UI
* removed the weird '()' empty name component in .json exporters
* cleaned up the namespace colour list widget in options->tag presentation. it now has proper add and delete buttons
* refactored the colour picker button significantly and moved and merged its old wx patch code into the main object
* the duplicate filter handles 'cannot rewind' errors better, including if the first pair is no longer viewable
* pretty sure I fixed a long-time stupid hang in the unit tests that appeared occasionally after a 'favicon' fech test. it was due to a previous network engine shutdown test applying too broadly to test objects
* cleaned up some edge cases in the 'which account added this file/mapping to the server?' tech, where it might have been possible, when looking up deleted content, to get another janitor account (i.e. who deleted the content), although I am pretty sure this situation was never possible to actually start in UI. if I add 'who deleted this?' tech in future, it'll be a separate specific call
* cleaned up some specifically 'Qt6' references in the build script. the build requirements.txts and spec files are also collapsed down, with old Qt5 versions removed
* filled out some incomplete abstract class definitions

View File

@ -34,6 +34,31 @@
<div class="content">
<h1 id="changelog"><a href="#changelog">changelog</a></h1>
<ul>
<li>
<h2 id="version_515"><a href="#version_515">version 515</a></h2>
<ul>
<li><h3>related tags</h3></li>
<li>I worked on last week's related tags algorithm test, bringing it up to usable standard. the old buttons now use the new algorithm exclusively. all users now get 'related tags' showing in manage tags by default (if you don't like it, you can turn it off under _options->tag suggestions_)</li>
<li>the new algorithm has new cancel tech and does a 'work for 600ms' kind of deal, like the old system, and the last-minute blocks from last week are gone--it will search as much as it has time for, including partial results. it also won't lag you out for thirty seconds (unless you tell it to in the options). it searches tags with low count first, so don't worry if it doesn't get to everything--'1girl' usually doesn't have a huge amount extra to offer once everything else has run</li>
<li>it also uses 'hydev actually thought about this' statistical sampling tech to work massively faster on larger-count tags at the cost of some variance in rank and the odd false positive (considered sufficiently related when it actually shouldn't meet the threshold) nearer the bottom end of the tags result list</li>
<li>rather than 'new 1' and 'new 2', there is now an on/off button for searching your local files or all known files on tag repositories. 'all known files' = great results, but very slow, which the tooltip explains</li>
<li>there's also a new status label that will tell you when it is searching and how well the search went (e.g. '12/51 tags searched fully in 459ms')</li>
<li>I also added the 'quick' search button back in, since we can now repeat searches for just selections of tags</li>
<li>I fixed a couple typos in the algorthim that were messing some results</li>
<li>in the manage tags dialog, if you have the suggested tag panels 'side-to-side', they now go in named boxes</li>
<li>in the manage tags dialog, if you have suggested tag panels in a notebook, 'related tags' will only refresh its search on a media change event (including dialog initialisation) when it is the selected page. it won't lag you from the background!</li>
<li>options->tag suggestions now lets you pick which notebook'd tag suggestions page you want to show by default. this defaults to 'related'</li>
<li>I have more plans here. these related tags results are very cachable, so that's an obvious next step to speed up results, and when I have done some other long-term tag improvements elsewhere in the program, I'll be able to quickly filter out unhelpful sibling and parent suggestions. more immediately, I think we'll want some options for namespace weighting (e.g. 'series:' tags' suggestions could have higher rank than 'smile'), so we can tune things a bit</li>
<li><h3>misc</h3></li>
<li>the 'open externally' canvas widget, which shows any available thumbnail of the flash or psd or whatever, now sizes itself correctly and draws the thumbnail nicely if you set the new thumbnail supersampling option to >100%. if your thumbnail is the wrong size (and probably in a queue to be regenerated soon), I _think_ it'll still make the window too big/small, but it'll draw the thumbnail to fit</li>
<li>if a tag content update comes in with an invalid tag (such as could happen with sidecars recently), the client now heals better. the bad tag is corrected live in more places, and this should be propagated to the UI. if you got a warning about 'you have invalid tags in view' recently but running the routine found no problems, please reboot, and I think you'll be fixed. I'm pretty sure the database wasn't being damaged at all here (it has cleaning safeguards, so it _shouldn't_ be possible to actually save bad tags)--it was just a thing to do with the UI not being told of the cleaned tag, and it shouldn't happen again. thank you for the reports! (issue #1324)</li>
<li>export folders and the file maintenance dialog no longer apply the implicit system:limit (defaults to max 10k files) to their searches!</li>
<li>old OR predicates that you load with saved searches and similar should now always have alphebetised components, and if you double-click them to remove them, they will now clear correctly (previously, they were doing something similar to the recent filetype problem, where instead of recognising themselves and deleting, they would instead duplicate a normalised (sorted) copy of themselves)</li>
<li>thanks to a user, updated the recently note-and-ai-updated pixiv parser again to grab the canonical pixiv URL and translated tags, if present</li>
<li>thanks to a user, updated the sankaku parser to grab some more tags</li>
<li>the file location context and tag context buttons under tag autocompletes now put menu separators between each type of file/tag service in their menus. for basic users, this'll be a separator for every row, but for advanced users with multiple local domains, it will help categorise the list a bit</li>
</ul>
</li>
<li>
<h2 id="version_514"><a href="#version_514">version 514</a></h2>
<ul>

View File

@ -133,7 +133,7 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
self._dictionary[ 'booleans' ][ 'activate_window_on_tag_search_page_activation' ] = False
self._dictionary[ 'booleans' ][ 'show_related_tags' ] = False
self._dictionary[ 'booleans' ][ 'show_related_tags' ] = True
self._dictionary[ 'booleans' ][ 'show_file_lookup_script_tags' ] = False
self._dictionary[ 'booleans' ][ 'freeze_message_manager_when_mouse_on_other_monitor' ] = False
@ -559,6 +559,7 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
self._dictionary[ 'strings' ][ 'has_audio_label' ] = '\U0001F50A'
self._dictionary[ 'strings' ][ 'has_duration_label' ] = ' \u23F5 '
self._dictionary[ 'strings' ][ 'discord_dnd_filename_pattern' ] = '{hash}'
self._dictionary[ 'strings' ][ 'default_suggested_tags_notebook_page' ] = 'related'
self._dictionary[ 'string_list' ] = {}

View File

@ -353,7 +353,7 @@ HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIAL
class FileSystemPredicates( object ):
def __init__( self, system_predicates: typing.Collection[ "Predicate" ], apply_implicit_limit = True ):
def __init__( self, system_predicates: typing.Collection[ "Predicate" ] ):
self._has_system_everything = False
@ -1765,7 +1765,7 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
serialisable_or_predicates = serialisable_value
self._value = tuple( HydrusSerialisable.CreateFromSerialisableTuple( serialisable_or_predicates ) )
self._value = tuple( sorted( HydrusSerialisable.CreateFromSerialisableTuple( serialisable_or_predicates ), key = lambda p: HydrusTags.ConvertTagToSortable( p.ToString() ) ) )
else:

View File

@ -1,6 +1,7 @@
import collections
import hashlib
import itertools
import math
import os
import random
import sqlite3
@ -5068,97 +5069,7 @@ class DB( HydrusDB.HydrusDB ):
return sorted_recent_tags
def _GetRelatedTags( self, service_key, skip_hash, search_tags, max_results, max_time_to_take ):
stop_time_for_finding_files = HydrusData.GetNowPrecise() + ( max_time_to_take / 2 )
stop_time_for_finding_tags = HydrusData.GetNowPrecise() + ( max_time_to_take / 2 )
service_id = self.modules_services.GetServiceId( service_key )
skip_hash_id = self.modules_hashes_local_cache.GetHashId( skip_hash )
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = ClientDBMappingsStorage.GenerateMappingsTableNames( service_id )
tag_ids = [ self.modules_tags.GetTagId( tag ) for tag in search_tags ]
random.shuffle( tag_ids )
hash_ids_counter = collections.Counter()
with self._MakeTemporaryIntegerTable( tag_ids, 'tag_id' ) as temp_table_name:
# temp tags to mappings
cursor = self._Execute( 'SELECT hash_id FROM {} CROSS JOIN {} USING ( tag_id );'.format( temp_table_name, current_mappings_table_name ) )
cancelled_hook = lambda: HydrusData.TimeHasPassedPrecise( stop_time_for_finding_files )
for ( hash_id, ) in HydrusDB.ReadFromCancellableCursor( cursor, 128, cancelled_hook = cancelled_hook ):
hash_ids_counter[ hash_id ] += 1
if skip_hash_id in hash_ids_counter:
del hash_ids_counter[ skip_hash_id ]
#
if len( hash_ids_counter ) == 0:
return []
# this stuff is often 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1.....
# the 1 stuff often produces large quantities of the same very popular tag, so your search for [ 'eva', 'female' ] will produce 'touhou' because so many 2hu images have 'female'
# so we want to do a 'soft' intersect, only picking the files that have the greatest number of shared search_tags
# this filters to only the '2' results, which gives us eva females and their hair colour and a few choice other popular tags for that particular domain
[ ( gumpf, largest_count ) ] = hash_ids_counter.most_common( 1 )
hash_ids = [ hash_id for ( hash_id, current_count ) in hash_ids_counter.items() if current_count > largest_count * 0.8 ]
counter = collections.Counter()
random.shuffle( hash_ids )
for hash_id in hash_ids:
for tag_id in self._STI( self._Execute( 'SELECT tag_id FROM ' + current_mappings_table_name + ' WHERE hash_id = ?;', ( hash_id, ) ) ):
counter[ tag_id ] += 1
if HydrusData.TimeHasPassedPrecise( stop_time_for_finding_tags ):
break
#
for tag_id in tag_ids:
if tag_id in counter:
del counter[ tag_id ]
results = counter.most_common( max_results )
inclusive = True
pending_count = 0
tag_ids_to_full_counts = { tag_id : ( current_count, None, pending_count, None ) for ( tag_id, current_count ) in results }
predicates = self.modules_tag_display.GeneratePredicatesFromTagIdsAndCounts( ClientTags.TAG_DISPLAY_STORAGE, service_id, tag_ids_to_full_counts, inclusive )
return predicates
def _GetRelatedTagsNewOneTag( self, tag_display_type, file_service_id, tag_service_id, search_tag_id ):
def _GetRelatedTagCountsForOneTag( self, tag_display_type, file_service_id, tag_service_id, search_tag_id, max_num_files_to_search, stop_time_for_finding_results = None ) -> typing.Tuple[ collections.Counter, bool ]:
# a user provided the basic idea here
@ -5166,10 +5077,13 @@ class DB( HydrusDB.HydrusDB ):
# specifying namespace is critical to increase search speed, otherwise we actually are searching all tags for tags
# we also call this with single specific file domains to keep things fast
# also this thing searches in fixed file domain to get fast
# this table selection is hacky as anything, but simpler than GetMappingAndTagTables for now
# this would be an ideal location to have a normal-acting cache of results
# a two-table-per service-cross-reference thing with cache entry + a creation timestamp and the actual mappings. invalidate on age or some tag changes I guess
# then here we'll poll the search tag to give results, invalidate old ones, then populate as needed and return
# only cache what you finish though!
mappings_table_names = []
if file_service_id == self.modules_services.combined_file_service_id:
@ -5194,31 +5108,73 @@ class DB( HydrusDB.HydrusDB ):
results = collections.Counter()
tags_table_name = self.modules_tag_search.GetTagsTableName( file_service_id, tag_service_id )
# note we used to filter by namespace here and needed the tags table, but no longer. might come back one day, but might be more trouble than it is worth
# tags_table_name = self.modules_tag_search.GetTagsTableName( file_service_id, tag_service_id )
# while this searches pending and current tags, it does not cross-reference current and pending on the same file, oh well!
cancelled_hook = None
if stop_time_for_finding_results is not None:
def cancelled_hook():
return HydrusData.TimeHasPassedPrecise( stop_time_for_finding_results )
results_dict = collections.Counter()
we_stopped_early = False
for mappings_table_name in mappings_table_names:
search_predicate = 'hash_id IN ( SELECT hash_id FROM {} WHERE tag_id = {} )'.format( mappings_table_name, search_tag_id )
# if we do the straight-up 'SELECT tag_id, COUNT( * ) FROM {} WHERE hash_id IN ( SELECT hash_id FROM {} WHERE tag_id = {} ) GROUP BY tag_id;
# then this is not easily cancellable. since it is working by hash_id, it doesn't know any counts until it knows all of them and is finished
# trying to break it into two with a temp integer table runs into the same issue or needs us to pull a bunch of ( tag_id, 1 ) counts
# since we'll be grabbing tag_ids with 1 count anyway for cancel tech, let's count them ourselves and trust the overhead isn't killer
query = 'SELECT tag_id, COUNT( * ) FROM {} CROSS JOIN {} USING ( tag_id ) WHERE {} GROUP BY subtag_id;'.format( mappings_table_name, tags_table_name, search_predicate )
# UPDATE: I added the ORDER BY RANDOM() LIMIT 1000 here as a way to better sample. We don't care about all results, we care about samples
# Unfortunately I think I have to do the RANDOM, since non-random search will bias the sample to early files etc...
# However this reduces the search space significantly, although I have to wangle some other numbers in the parent method
results.update( dict( self._Execute( query ).fetchall() ) )
# this may cause laggy cancel tech since I think the whole order by random has to be done before any results will come back, which for '1girl' is going to be millions of rows...
# we'll see how it goes
search_predicate = 'hash_id IN ( SELECT hash_id FROM {} WHERE tag_id = {} ORDER BY RANDOM() LIMIT {} )'.format( mappings_table_name, search_tag_id, max_num_files_to_search )
query = 'SELECT tag_id FROM {} WHERE {};'.format( mappings_table_name, search_predicate )
cursor = self._Execute( query )
loop_of_results = self._STI( HydrusDB.ReadFromCancellableCursor( cursor, 1024, cancelled_hook = cancelled_hook ) )
# counter can just take a list of gubbins like this
results_dict.update( loop_of_results )
if cancelled_hook():
we_stopped_early = True
break
return results
return ( results_dict, we_stopped_early )
def _GetRelatedTagsNew( self, file_service_key, tag_service_key, search_tags, max_results = 100, concurrence_threshold = 0.04, total_search_tag_count_threshold = 25000 ):
def _GetRelatedTags( self, file_service_key, tag_service_key, search_tags, max_time_to_take = 0.5, max_results = 100, concurrence_threshold = 0.04 ):
num_tags_searched = 0
num_tags_to_search = 0
stop_time_for_finding_results = HydrusData.GetNowPrecise() + ( max_time_to_take * 0.85 )
# a user provided the basic idea here
if len( search_tags ) == 0:
return [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, value = 'no search tags to work with!' ) ]
return ( num_tags_searched, num_tags_to_search, [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, value = 'no search tags to work with!' ) ] )
tag_display_type = ClientTags.TAG_DISPLAY_ACTUAL
@ -5242,73 +5198,82 @@ class DB( HydrusDB.HydrusDB ):
#
search_tags = set()
# two things here:
# 1
# we don't really want to use '1girl' and friends as search tags here, since the search domain is so huge
# so, we go for the smallest count tags first. they have interesting suggestions
# 2
# namespaces tend to be richer as suggestions, so we'll do them first
for ( search_tag_id, count ) in search_tag_ids_to_total_counts.items():
search_tag_ids_flat_sorted_ascending = sorted( search_tag_ids_to_total_counts.items(), key = lambda row: ( HydrusTags.IsUnnamespaced( search_tag_ids_to_search_tags[ row[0] ] ), row[1] ) )
search_tags_sorted_ascending = []
for ( search_tag_id, count ) in search_tag_ids_flat_sorted_ascending:
# pending only
if count == 0:
# I had a negative count IRL, must have been some crazy miscount, caused heaps of trouble with later square root causing imaginary numbers!!!
# Having count 0 here is only _supposed_ to happen if the user is asking about stuff they just pended in the dialog now, before hitting ok, or if they are searching across domains
# _Or_ if they are tagging files they don't have and searching on local domain
# However I saw '10 tags had no related data' on a local file on dev machine with (+1) pending tags!!!
# It was evidence of some busted A/C caches, it seems. ghost tags that were present on the cache but not the base mappings tables!
if count <= 0:
continue
search_tags.add( search_tag_ids_to_search_tags[ search_tag_id ] )
search_tags_sorted_ascending.append( search_tag_ids_to_search_tags[ search_tag_id ] )
if len( search_tags ) == 0:
num_tags_to_search = len( search_tags_sorted_ascending )
if num_tags_to_search == 0:
return [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, value = 'not enough data in search domain' ) ]
# all have count 0
return ( num_tags_searched, num_tags_to_search, [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, value = 'no related tags found!' ) ] )
#
search_tag_ids_flat_sorted_ascending = sorted( search_tag_ids_to_total_counts.items(), key = lambda row: row[1] )
search_tags = set()
total_count = 0
# TODO: I think I would rather rework this into a time threshold thing like the old related tags stuff.
# so, should ditch the culling and instead make all the requests cancellable. just keep searching until we are out of time, then put results together
# TODO: Another option as I vacillate on 'all my files' vs 'all known files' would be to incorporate that into the search timer
# do all my files first, then replace that with all known files results until we run out of time (only do this for repositories)
# we don't really want to use '1girl' and friends as search tags here, since the search domain is so huge
# so, we go for the smallest count tags first. they have interesting suggestions
# searching all known files is gonkmode, so we curtail our max search size
if file_service_key == CC.COMBINED_FILE_SERVICE_KEY:
total_search_tag_count_threshold /= 25
for ( search_tag_id, count ) in search_tag_ids_flat_sorted_ascending:
# we don't want the total domain to be too large either. death by a thousand cuts
if total_count + count > total_search_tag_count_threshold:
break
total_count += count
search_tags.add( search_tag_ids_to_search_tags[ search_tag_id ] )
if len( search_tags ) == 0:
return [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, value = 'search domain too big' ) ]
# max_num_files_to_search = 1000
# 1000 gives us a decent sample mate, no matter the population size
# we can probably go lower than this, or rather base it dynamically on the concurrence_threshold.
# if a tag t has to have 0.04 out of n to match, then can't we figure out a sample size n that is 97% likely to catch t>=1 for the least likely qualifying t?
#
# hydev attempts to do stats here, potential ruh roh
# what sample size do we need to have 97% liklihood of getting at least one of the least likely (how many draws of 1-in-25 to get at least one hit)
# this is cumulative binomial probability, maybe, with success chance 0.04 and result X >= 1. what's the n for P(X>=1) = 0.97?
# this is probably wrong and stupid, but whatever
# for difficult values of X I think we need some inverse cumulative distribution function, which is beyond my expertise
# but isn't p(X>=1) the same as 1 - P(X=0)? and chance of none happening is just (24/25)^n
# (24/25)^n = 0.03
# hydev last did this for real 19 years ago, but:
# n = log( 0.03 ) / log( 0.96 ) = 86
# 143 for 0.997
# actually sounds about right?????
# a secondary problem here is when we correct our scores later on, we've got some low 'count' counts causing some variance and spiky ranking, particularly at the bottom
# your virtual sampled x = 1.2 and 1.4 will be 1 or 2 swapped around at times, or higher like 4 and 5, and so will bump a bit
# also while we minimised false negatives, we get some 0.4 getting a hit and counting as 1 of course and then it counts as something, so we've introduced false positives, hooray
# to smooth them, we'll just multiple our n a bit. ideally we'd pick an x in P(X>=x) large enough that the granular steps reduce variance
# in the end we spent a bunch of brainpower rationalising a guess of 1,000 down to 429, but at least there's a framework here to iterate on
desired_confidence = 0.997
chance_of_success = concurrence_threshold
max_num_files_to_search = int( math.ceil( math.log( 1 - desired_confidence ) / math.log( 1 - chance_of_success ) ) )
magical_later_multiplication_smoothing_coefficient = 3
max_num_files_to_search *= magical_later_multiplication_smoothing_coefficient
search_tag_ids_to_tag_ids_to_matching_counts = {}
for search_tag in search_tags:
for search_tag in search_tags_sorted_ascending:
search_tag_id = self.modules_tags_local_cache.GetTagId( search_tag )
tag_ids_to_matching_counts = self._GetRelatedTagsNewOneTag( tag_display_type, file_service_id, tag_service_id, search_tag_id )
( tag_ids_to_matching_counts, it_stopped_early ) = self._GetRelatedTagCountsForOneTag( tag_display_type, file_service_id, tag_service_id, search_tag_id, max_num_files_to_search, stop_time_for_finding_results = stop_time_for_finding_results )
if search_tag_id in tag_ids_to_matching_counts:
@ -5317,15 +5282,17 @@ class DB( HydrusDB.HydrusDB ):
search_tag_ids_to_tag_ids_to_matching_counts[ search_tag_id ] = tag_ids_to_matching_counts
if it_stopped_early:
break
num_tags_searched += 1
#
# ok we have a bunch of counts here for different search tags, so let's figure out some normalised scores and merge them all
#
# the master score is: number matching mappings found / square_root( suggestion_tag_count * search_tag_count )
#
# I don't really know what this *is*, but the user did it and it seems to make nice scores, so hooray
# the dude said it was arbitrary and could do with tuning, so we'll see how it goes
all_tag_ids = set()
@ -5345,6 +5312,13 @@ class DB( HydrusDB.HydrusDB ):
tag_ids_to_scores = collections.Counter()
# the master score is: number matching mappings found / square_root( suggestion_tag_count * search_tag_count )
#
# the dude said it was mostly arbitrary but came from, I think, P-TAG: Large Scale Automatic Generation of Personalized Annotation TAGs for the Web
# he said it could do with tuning, so we'll see how it goes, but overall I am happy with it
#
# UPDATE: Adding the 'max_num_files_to_search' thing above skews the score here, so we need to adjust it so our score and concurrence thresholds still work!
for ( search_tag_id, tag_ids_to_matching_counts ) in search_tag_ids_to_tag_ids_to_matching_counts.items():
if search_tag_id not in tag_ids_to_total_counts:
@ -5354,23 +5328,35 @@ class DB( HydrusDB.HydrusDB ):
search_tag_count = tag_ids_to_total_counts[ search_tag_id ]
matching_count_multiplier = 1.0
if search_tag_count > max_num_files_to_search:
# had we searched everything, how much bigger would the results probably be?
matching_count_multiplier = search_tag_count / max_num_files_to_search
search_tag_is_unnamespaced = HydrusTags.IsUnnamespaced( search_tag_ids_to_search_tags[ search_tag_id ] )
for ( tag_id, matching_count ) in tag_ids_to_matching_counts.items():
for ( suggestion_tag_id, suggestion_matching_count ) in tag_ids_to_matching_counts.items():
if matching_count / search_tag_count < concurrence_threshold:
suggestion_matching_count *= matching_count_multiplier
if suggestion_matching_count / search_tag_count < concurrence_threshold:
# this result didn't turn up enough to be relevant
continue
if tag_id not in tag_ids_to_total_counts:
if suggestion_tag_id not in tag_ids_to_total_counts:
# probably a damaged A/C cache
continue
suggestion_tag_count = tag_ids_to_total_counts[ tag_id ]
suggestion_tag_count = tag_ids_to_total_counts[ suggestion_tag_id ]
score = matching_count / ( ( abs( suggestion_tag_count ) * abs( search_tag_count ) ) ** 0.5 )
score = suggestion_matching_count / ( ( abs( suggestion_tag_count ) * abs( search_tag_count ) ) ** 0.5 )
# sophisticated hydev score-tuning
if search_tag_is_unnamespaced:
@ -5378,7 +5364,7 @@ class DB( HydrusDB.HydrusDB ):
score /= 3
tag_ids_to_scores[ tag_id ] += float( score )
tag_ids_to_scores[ suggestion_tag_id ] += float( score )
@ -5395,7 +5381,7 @@ class DB( HydrusDB.HydrusDB ):
predicates = self.modules_tag_display.GeneratePredicatesFromTagIdsAndCounts( tag_display_type, tag_service_id, tag_ids_to_full_counts, inclusive )
return predicates
return ( num_tags_searched, num_tags_to_search, predicates )
def _GetRepositoryThumbnailHashesIDoNotHave( self, service_key ):
@ -6797,10 +6783,21 @@ class DB( HydrusDB.HydrusDB ):
try:
potentially_dirty_tag = tag
tag = HydrusTags.CleanTag( potentially_dirty_tag )
if tag != potentially_dirty_tag:
content_update.SetRow( ( tag, hashes ) )
tag_id = self.modules_tags.GetTagId( tag )
except HydrusExceptions.TagSizeException:
content_update.SetRow( ( 'bad tag', set() ) )
continue
@ -7619,7 +7616,6 @@ class DB( HydrusDB.HydrusDB ):
elif action == 'services': result = self.modules_services.GetServices( *args, **kwargs )
elif action == 'similar_files_maintenance_status': result = self.modules_similar_files.GetMaintenanceStatus( *args, **kwargs )
elif action == 'related_tags': result = self._GetRelatedTags( *args, **kwargs )
elif action == 'related_tags_new': result = self._GetRelatedTagsNew( *args, **kwargs )
elif action == 'tag_display_application': result = self.modules_tag_display.GetApplication( *args, **kwargs )
elif action == 'tag_display_maintenance_status': result = self._CacheTagDisplayGetApplicationStatusNumbers( *args, **kwargs )
elif action == 'tag_parents': result = self.modules_tag_parents.GetTagParents( *args, **kwargs )
@ -11113,6 +11109,64 @@ class DB( HydrusDB.HydrusDB ):
if version == 514:
try:
new_options = self.modules_serialisable.GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_CLIENT_OPTIONS )
if not new_options.GetBoolean( 'show_related_tags' ):
new_options.SetBoolean( 'show_related_tags', True )
self.modules_serialisable.SetJSONDump( new_options )
message = 'Hey, I made it so your manage tags dialog shows the updated "related tags" suggestion column (showing it is the new default). If you do not like it, turn it off again under _options->tag suggestions_, thanks!'
self.pub_initial_message( message )
except Exception as e:
HydrusData.PrintException( e )
message = 'Trying to update your options to show related tags failed! Please let hydrus dev know!'
self.pub_initial_message( message )
try:
domain_manager = self.modules_serialisable.GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER )
domain_manager.Initialise()
#
domain_manager.OverwriteDefaultParsers( [
'sankaku file page parser',
'pixiv file page api parser'
] )
#
domain_manager.TryToLinkURLClassesAndParsers()
#
self.modules_serialisable.SetJSONDump( domain_manager )
except Exception as e:
HydrusData.PrintException( e )
message = 'Trying to update some downloader objects failed! Please let hydrus dev know!'
self.pub_initial_message( message )
self._controller.frame_splash_status.SetTitleText( 'updated db to v{}'.format( HydrusData.ToHumanInt( version + 1 ) ) )
self._Execute( 'UPDATE version SET version = ?;', ( version + 1, ) )

View File

@ -420,7 +420,7 @@ class ExportFolder( HydrusSerialisable.SerialisableBaseNamed ):
def _DoExport( self ):
query_hash_ids = HG.client_controller.Read( 'file_query_ids', self._file_search_context )
query_hash_ids = HG.client_controller.Read( 'file_query_ids', self._file_search_context, apply_implicit_limit = False )
media_results = []

View File

@ -3778,6 +3778,13 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._suggested_tags_layout.addItem( 'notebook', 'notebook' )
self._suggested_tags_layout.addItem( 'side-by-side', 'columns' )
self._default_suggested_tags_notebook_page = ClientGUICommon.BetterChoice( suggested_tags_panel )
for item in [ 'favourites', 'related', 'file_lookup_scripts', 'recent' ]:
self._default_suggested_tags_notebook_page.addItem( item, item )
suggest_tags_panel_notebook = QW.QTabWidget( suggested_tags_panel )
#
@ -3842,6 +3849,10 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._suggested_tags_layout.SetValue( self._new_options.GetNoneableString( 'suggested_tags_layout' ) )
self._default_suggested_tags_notebook_page.SetValue( self._new_options.GetString( 'default_suggested_tags_notebook_page' ) )
#
self._show_related_tags.setChecked( self._new_options.GetBoolean( 'show_related_tags' ) )
self._related_tags_search_1_duration_ms.setValue( self._new_options.GetInteger( 'related_tags_search_1_duration_ms' ) )
@ -3871,7 +3882,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
rows = []
rows.append( ( 'Show related tags: ', self._show_related_tags ) )
rows.append( ( 'Initial search duration (ms): ', self._related_tags_search_1_duration_ms ) )
rows.append( ( 'Initial/Quick search duration (ms): ', self._related_tags_search_1_duration_ms ) )
rows.append( ( 'Medium search duration (ms): ', self._related_tags_search_2_duration_ms ) )
rows.append( ( 'Thorough search duration (ms): ', self._related_tags_search_3_duration_ms ) )
@ -3879,7 +3890,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
desc = 'This will search the database for tags statistically related to what your files already have.'
QP.AddToLayout( panel_vbox, ClientGUICommon.BetterStaticText(suggested_tags_related_panel,desc), CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( panel_vbox, ClientGUICommon.BetterStaticText( suggested_tags_related_panel, desc ), CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( panel_vbox, gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
suggested_tags_related_panel.setLayout( panel_vbox )
@ -3922,10 +3933,11 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
rows.append( ( 'Width of suggested tags columns: ', self._suggested_tags_width ) )
rows.append( ( 'Column layout: ', self._suggested_tags_layout ) )
rows.append( ( 'Default notebook page: ', self._default_suggested_tags_notebook_page ) )
gridbox = ClientGUICommon.WrapInGrid( suggested_tags_panel, rows )
desc = 'The manage tags dialog can provide several kinds of tag suggestions. For simplicity, most are turned off by default.'
desc = 'The manage tags dialog can provide several kinds of tag suggestions.'
suggested_tags_panel.Add( ClientGUICommon.BetterStaticText( suggested_tags_panel, desc ), CC.FLAGS_EXPAND_PERPENDICULAR )
suggested_tags_panel.Add( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
@ -3942,10 +3954,19 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
#
self._suggested_favourites_services.currentIndexChanged.connect( self.EventSuggestedFavouritesService )
self._suggested_tags_layout.currentIndexChanged.connect( self._NotifyLayoutChanged )
self._NotifyLayoutChanged()
self.EventSuggestedFavouritesService( None )
def _NotifyLayoutChanged( self ):
enable_default_page = self._suggested_tags_layout.GetValue() == 'notebook'
self._default_suggested_tags_notebook_page.setEnabled( enable_default_page )
def _SaveCurrentSuggestedFavourites( self ):
if self._current_suggested_favourites_service is not None:
@ -3982,6 +4003,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._new_options.SetInteger( 'suggested_tags_width', self._suggested_tags_width.value() )
self._new_options.SetNoneableString( 'suggested_tags_layout', self._suggested_tags_layout.GetValue() )
self._new_options.SetString( 'default_suggested_tags_notebook_page', self._default_suggested_tags_notebook_page.GetValue() )
self._SaveCurrentSuggestedFavourites()
for ( service_key, favourites ) in list(self._suggested_favourites_dict.items()):

View File

@ -2741,7 +2741,7 @@ class ReviewFileMaintenance( ClientGUIScrolledPanels.ReviewPanel ):
def work_callable():
query_hash_ids = HG.client_controller.Read( 'file_query_ids', file_search_context )
query_hash_ids = HG.client_controller.Read( 'file_query_ids', file_search_context, apply_implicit_limit = False )
return query_hash_ids
@ -2781,7 +2781,7 @@ class ReviewFileMaintenance( ClientGUIScrolledPanels.ReviewPanel ):
def work_callable():
query_hash_ids = HG.client_controller.Read( 'file_query_ids', file_search_context )
query_hash_ids = HG.client_controller.Read( 'file_query_ids', file_search_context, apply_implicit_limit = False )
return query_hash_ids
@ -2808,7 +2808,7 @@ class ReviewFileMaintenance( ClientGUIScrolledPanels.ReviewPanel ):
def work_callable():
query_hash_ids = HG.client_controller.Read( 'file_query_ids', file_search_context )
query_hash_ids = HG.client_controller.Read( 'file_query_ids', file_search_context, apply_implicit_limit = False )
return query_hash_ids

View File

@ -34,29 +34,33 @@ def FilterSuggestedPredicatesForMedia( predicates: typing.Sequence[ ClientSearch
return predicates
def FilterSuggestedTagsForMedia( tags: typing.Sequence[ str ], medias: typing.Collection[ ClientMedia.Media ], service_key: bytes ) -> typing.List[ str ]:
# TODO: figure out a nice way to filter out siblings here
# maybe have to wait for when tags always know their siblings
# then we could also filter out worse/better siblings of the same count
num_media = len( medias )
tags_filtered_set = set( tags )
useful_tags_set = set( tags )
( current_tags_to_count, deleted_tags_to_count, pending_tags_to_count, petitioned_tags_to_count ) = ClientMedia.GetMediasTagCount( medias, service_key, ClientTags.TAG_DISPLAY_STORAGE )
current_tags_to_count.update( pending_tags_to_count )
# TODO: figure out a nicer way to filter out siblings and parents here
# maybe have to wait for when tags always know their siblings
# then we could also filter out worse/better siblings of the same count
# this is a sync way for now:
# db_tags_to_ideals_and_parents = HG.client_controller.Read( 'tag_display_decorators', service_key, tags_to_lookup )
for ( tag, count ) in current_tags_to_count.items():
if count == num_media:
tags_filtered_set.discard( tag )
useful_tags_set.discard( tag )
tags_filtered = [ tag for tag in tags if tag in tags_filtered_set ]
tags_filtered = [ tag for tag in tags if tag in useful_tags_set ]
return tags_filtered
@ -334,7 +338,7 @@ class RelatedTagsPanel( QW.QWidget ):
self._last_fetched_predicates = []
self._have_fetched = False
self._have_done_search_with_this_media = False
self._selected_tags = set()
@ -342,8 +346,18 @@ class RelatedTagsPanel( QW.QWidget ):
vbox = QP.VBoxLayout()
self._status_label = ClientGUICommon.BetterStaticText( self, label = 'ready' )
self._just_do_local_files = ClientGUICommon.OnOffButton( self, on_label = 'just for my files', off_label = 'for all known files', start_on = True )
self._just_do_local_files.setToolTip( 'Select how big the search is. Searching across all known files on a repository produces high quality results but takes a long time.' )
tt = 'If you select some tags, this will search using only those as reference!'
self._button_1 = QW.QPushButton( 'quick', self )
self._button_1.clicked.connect( self.RefreshQuick )
self._button_1.setMinimumWidth( 30 )
self._button_1.setToolTip( tt )
self._button_2 = QW.QPushButton( 'medium', self )
self._button_2.clicked.connect( self.RefreshMedium )
self._button_2.setMinimumWidth( 30 )
@ -354,38 +368,21 @@ class RelatedTagsPanel( QW.QWidget ):
self._button_3.setMinimumWidth( 30 )
self._button_3.setToolTip( tt )
self._button_new = QW.QPushButton( 'new 1', self )
self._button_new.clicked.connect( self.RefreshNew )
self._button_new.setMinimumWidth( 30 )
tt = 'Please test this! This uses the new statistical method and searches your local files\' tags. Should be pretty fast, but its search domain is limited.' + os.linesep * 2 + 'Hydev thinks this mode sucks for the PTR, so let him know if it is actually works ok there.'
self._button_new.setToolTip( tt )
self._button_new_2 = QW.QPushButton( 'new 2', self )
self._button_new_2.clicked.connect( self.RefreshNew2 )
self._button_new_2.setMinimumWidth( 30 )
tt = 'Please test this! This uses the new statistical method and searches all the service\'s tags. May search slow and will not get results from large-count tags.' + os.linesep * 2 + 'Hydev wants to use this in the end, so let him know if it is too laggy.'
self._button_new_2.setToolTip( tt )
if len( self._media ) > 1:
self._button_2.setVisible( False )
self._button_3.setVisible( False )
if HG.client_controller.services_manager.GetServiceType( self._service_key ) == HC.LOCAL_TAG:
self._button_new_2.setVisible( False )
self._just_do_local_files.setVisible( False )
self._related_tags = ListBoxTagsSuggestionsRelated( self, service_key, activate_callable )
button_hbox = QP.HBoxLayout()
QP.AddToLayout( button_hbox, self._button_1, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
QP.AddToLayout( button_hbox, self._button_2, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
QP.AddToLayout( button_hbox, self._button_3, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
QP.AddToLayout( button_hbox, self._button_new, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
QP.AddToLayout( button_hbox, self._button_new_2, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
QP.AddToLayout( vbox, self._status_label, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._just_do_local_files, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, button_hbox, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._related_tags, CC.FLAGS_EXPAND_BOTH_WAYS )
@ -394,111 +391,97 @@ class RelatedTagsPanel( QW.QWidget ):
self._related_tags.mouseActivationOccurred.connect( self.mouseActivationOccurred )
def _FetchRelatedTags( self, max_time_to_take ):
def _FetchRelatedTagsNew( self, max_time_to_take ):
def do_it( service_key, hash, search_tags, max_results, max_time_to_take ):
def do_it( file_service_key, tag_service_key, search_tags ):
def qt_code( predicates ):
def qt_code( predicates, num_done, num_to_do, total_time_took ):
if not self or not QP.isValid( self ):
return
if num_to_do == len( search_tags ):
tags_s = 'tags'
else:
tags_s = 'tags ({} had no relations)'.format( HydrusData.ToHumanInt( len( search_tags ) - num_to_do ) )
if num_done == len( search_tags ):
num_done_s = 'Searched all {} {} in '.format( HydrusData.ToHumanInt( num_done ), tags_s )
elif num_done == num_to_do:
num_done_s = 'Searched {} {} in '.format( HydrusData.ToHumanInt( num_done ), tags_s )
else:
num_done_s = '{} {} searched fully in '.format( HydrusData.ConvertValueRangeToPrettyString( num_done, num_to_do ), tags_s )
label = '{}{}.'.format( num_done_s, HydrusData.TimeDeltaToPrettyTimeDelta( total_time_took ) )
self._status_label.setText( label )
self._last_fetched_predicates = predicates
self._UpdateTagDisplay()
self._have_fetched = True
self._have_done_search_with_this_media = True
predicates = HG.client_controller.Read( 'related_tags', service_key, hash, search_tags, max_results, max_time_to_take )
start_time = HydrusData.GetNowPrecise()
( num_done, num_to_do, predicates ) = HG.client_controller.Read( 'related_tags', file_service_key, tag_service_key, search_tags, max_time_to_take = max_time_to_take )
total_time_took = HydrusData.GetNowPrecise() - start_time
predicates = ClientSearch.SortPredicates( predicates )
QP.CallAfter( qt_code, predicates )
QP.CallAfter( qt_code, predicates, num_done, num_to_do, total_time_took )
self._related_tags.SetPredicates( [] )
if len( self._media ) > 1:
if len( self._selected_tags ) == 0:
search_tags = ClientMedia.GetMediasTags( self._media, self._service_key, ClientTags.TAG_DISPLAY_STORAGE, ( HC.CONTENT_STATUS_CURRENT, HC.CONTENT_STATUS_PENDING ) )
else:
search_tags = self._selected_tags
if len( search_tags ) == 0:
self._status_label.setVisible( False )
return
( m, ) = self._media
self._status_label.setVisible( True )
hash = m.GetHash()
self._status_label.setText( 'searching\u2026' )
# TODO: If user has some tags selected, use them instead
if len( self._selected_tags ) == 0:
search_tags = ClientMedia.GetMediasTags( self._media, self._service_key, ClientTags.TAG_DISPLAY_STORAGE, ( HC.CONTENT_STATUS_CURRENT, HC.CONTENT_STATUS_PENDING ) )
else:
search_tags = self._selected_tags
max_results = 100
HG.client_controller.CallToThread( do_it, self._service_key, hash, search_tags, max_results, max_time_to_take )
def _FetchRelatedTagsNew( self, file_service_key = None ):
def do_it( file_service_key, tag_service_key, search_tags ):
def qt_code( predicates ):
if not self or not QP.isValid( self ):
return
self._last_fetched_predicates = predicates
self._UpdateTagDisplay()
self._have_fetched = True
predicates = HG.client_controller.Read( 'related_tags_new', file_service_key, tag_service_key, search_tags )
predicates = ClientSearch.SortPredicates( predicates )
QP.CallAfter( qt_code, predicates )
self._related_tags.SetPredicates( [] )
if len( self._selected_tags ) == 0:
search_tags = ClientMedia.GetMediasTags( self._media, self._service_key, ClientTags.TAG_DISPLAY_STORAGE, ( HC.CONTENT_STATUS_CURRENT, HC.CONTENT_STATUS_PENDING ) )
else:
search_tags = self._selected_tags
if file_service_key is None:
if self._just_do_local_files.IsOn():
file_service_key = CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY
else:
file_service_key = CC.COMBINED_FILE_SERVICE_KEY
tag_service_key = self._service_key
HG.client_controller.CallToThread( do_it, file_service_key, tag_service_key, search_tags )
def _QuickSuggestedRelatedTags( self ):
max_time_to_take = self._new_options.GetInteger( 'related_tags_search_1_duration_ms' ) / 1000.0
self._FetchRelatedTags( max_time_to_take )
def _UpdateTagDisplay( self ):
predicates = FilterSuggestedPredicatesForMedia( self._last_fetched_predicates, self._media, self._service_key )
@ -506,28 +489,25 @@ class RelatedTagsPanel( QW.QWidget ):
self._related_tags.SetPredicates( predicates )
def RefreshQuick( self ):
max_time_to_take = self._new_options.GetInteger( 'related_tags_search_1_duration_ms' ) / 1000.0
self._FetchRelatedTagsNew( max_time_to_take )
def RefreshMedium( self ):
max_time_to_take = self._new_options.GetInteger( 'related_tags_search_2_duration_ms' ) / 1000.0
self._FetchRelatedTags( max_time_to_take )
self._FetchRelatedTagsNew( max_time_to_take )
def RefreshThorough( self ):
max_time_to_take = self._new_options.GetInteger( 'related_tags_search_3_duration_ms' ) / 1000.0
self._FetchRelatedTags( max_time_to_take )
def RefreshNew( self ):
self._FetchRelatedTagsNew()
def RefreshNew2( self ):
self._FetchRelatedTagsNew( file_service_key = CC.COMBINED_FILE_SERVICE_KEY )
self._FetchRelatedTagsNew( max_time_to_take )
def MediaUpdated( self ):
@ -535,18 +515,23 @@ class RelatedTagsPanel( QW.QWidget ):
self._UpdateTagDisplay()
def NotifyUserLooking( self ):
if not self._have_done_search_with_this_media:
self.RefreshQuick()
def SetMedia( self, media ):
self._media = media
if len( self._media ) == 1:
self._QuickSuggestedRelatedTags()
else:
self._related_tags.SetPredicates( [] )
self._status_label.setText( 'ready' )
self._related_tags.SetPredicates( [] )
self._have_done_search_with_this_media = False
def SetSelectedTags( self, tags ):
@ -842,11 +827,33 @@ class SuggestedTagsPanel( QW.QWidget ):
QP.AddToLayout( hbox, self._notebook, CC.FLAGS_EXPAND_BOTH_WAYS )
name_to_page_dict = {
'favourites' : self._favourite_tags,
'related' : self._related_tags,
'file_lookup_scripts' : self._file_lookup_script_tags,
'recent' : self._recent_tags
}
default_suggested_tags_notebook_page = self._new_options.GetString( 'default_suggested_tags_notebook_page' )
choice = name_to_page_dict.get( default_suggested_tags_notebook_page, None )
if choice is not None:
self._notebook.setCurrentWidget( choice )
self._notebook.currentChanged.connect( self._PageChanged )
elif layout_mode == 'columns':
for ( name, panel ) in panels:
QP.AddToLayout( hbox, panel, CC.FLAGS_EXPAND_PERPENDICULAR )
box_panel = ClientGUICommon.StaticBox( self, name )
box_panel.Add( panel, CC.FLAGS_EXPAND_BOTH_WAYS )
QP.AddToLayout( hbox, box_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
@ -856,6 +863,27 @@ class SuggestedTagsPanel( QW.QWidget ):
self.hide()
else:
self._PageChanged()
def _PageChanged( self ):
if self._notebook is None:
self._related_tags.NotifyUserLooking()
return
current_page = self._notebook.currentWidget()
if current_page == self._related_tags:
self._related_tags.NotifyUserLooking()
def MediaUpdated( self ):
@ -913,6 +941,8 @@ class SuggestedTagsPanel( QW.QWidget ):
self._related_tags.SetMedia( media )
self._PageChanged()
def SetSelectedTags( self, tags ):

View File

@ -195,7 +195,10 @@ def CalculateMediaContainerSize( media, device_pixel_ratio: float, zoom, show_ac
bounding_dimensions = HG.client_controller.options[ 'thumbnail_dimensions' ]
thumbnail_scale_type = HG.client_controller.new_options.GetInteger( 'thumbnail_scale_type' )
thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
# we want the device independant size here, not actual pixels, so want to keep this 100
#thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
thumbnail_dpr_percent = 100
( clip_rect, ( thumb_width, thumb_height ) ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( media.GetResolution(), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
@ -2732,6 +2735,13 @@ class OpenExternallyPanel( QW.QWidget ):
qt_pixmap = ClientRendering.GenerateHydrusBitmap( thumbnail_path, thumbnail_mime ).GetQtPixmap()
thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
if thumbnail_dpr_percent != 100:
qt_pixmap.setDevicePixelRatio( thumbnail_dpr_percent / 100 )
thumbnail_window = ClientGUICommon.BufferedWindowIcon( self, qt_pixmap )
QP.AddToLayout( vbox, thumbnail_window, CC.FLAGS_CENTER )
@ -2739,7 +2749,7 @@ class OpenExternallyPanel( QW.QWidget ):
m_text = HC.mime_string_lookup[ media.GetMime() ]
button = QW.QPushButton( 'open ' + m_text + ' externally', self )
button = QW.QPushButton( 'open {} externally'.format( m_text ), self )
button.setFocusPolicy( QC.Qt.NoFocus )

View File

@ -143,16 +143,25 @@ class LocationSearchContextButton( ClientGUICommon.BetterButton ):
menu = QW.QMenu()
last_seen_service_type = None
for service in services:
if last_seen_service_type is not None and last_seen_service_type != service.GetServiceType():
ClientGUIMenus.AppendSeparator( menu )
location_context = ClientLocation.LocationContext.STATICCreateSimple( service.GetServiceKey() )
ClientGUIMenus.AppendMenuItem( menu, service.GetName(), 'Change the current file domain to {}.'.format( service.GetName() ), self.SetValue, location_context )
last_seen_service_type = service.GetServiceType()
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( menu, 'multiple locations', 'Change the current file domain to something with multiple locations.', self._EditMultipleLocationContext )
ClientGUIMenus.AppendMenuItem( menu, 'multiple/deleted locations', 'Change the current file domain to something with multiple locations.', self._EditMultipleLocationContext )
CGC.core().PopupMenu( self, menu )

View File

@ -876,12 +876,21 @@ class TagContextButton( ClientGUICommon.BetterButton ):
menu = QW.QMenu()
last_seen_service_type = None
for service in services:
if last_seen_service_type is not None and last_seen_service_type != service.GetServiceType():
ClientGUIMenus.AppendSeparator( menu )
tag_context = ClientSearch.TagContext( service_key = service.GetServiceKey() )
ClientGUIMenus.AppendMenuItem( menu, service.GetName(), 'Change the current tag domain to {}.'.format( service.GetName() ), self.SetValue, tag_context )
last_seen_service_type = service.GetServiceType()
CGC.core().PopupMenu( self, menu )

View File

@ -722,11 +722,13 @@ class BufferedWindow( QW.QWidget ):
class BufferedWindowIcon( BufferedWindow ):
def __init__( self, parent, bmp, click_callable = None ):
def __init__( self, parent, pixmap: QG.QPixmap, click_callable = None ):
BufferedWindow.__init__( self, parent, size = bmp.size() )
device_independant_size = pixmap.size() / pixmap.devicePixelRatio()
self._bmp = bmp
BufferedWindow.__init__( self, parent, size = device_independant_size )
self._pixmap = pixmap
self._click_callable = click_callable
@ -738,13 +740,15 @@ class BufferedWindowIcon( BufferedWindow ):
painter.eraseRect( painter.viewport() )
if isinstance( self._bmp, QG.QImage ):
painter.setRenderHint( QG.QPainter.SmoothPixmapTransform, True ) # makes any scaling here due to jank thumbs look good
if isinstance( self._pixmap, QG.QImage ):
painter.drawImage( 0, 0, self._bmp )
painter.drawImage( self.rect(), self._pixmap )
else:
painter.drawPixmap( 0, 0, self._bmp )
painter.drawPixmap( self.rect(), self._pixmap )

View File

@ -84,7 +84,7 @@ options = {}
# Misc
NETWORK_VERSION = 20
SOFTWARE_VERSION = 514
SOFTWARE_VERSION = 515
CLIENT_API_VERSION = 41
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.0 KiB

After

Width:  |  Height:  |  Size: 3.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.4 KiB

After

Width:  |  Height:  |  Size: 3.0 KiB