RepoMirrors
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Version 393

This commit is contained in:
Hydrus Network Developer 2020-04-15 19:09:42 -05:00
parent 4b29d91b54
commit eabff1015b
38 changed files with 878 additions and 161 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

39
help/changelog.html
View File

@ -8,6 +8,45 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 393</h3></li>
<ul>
<li>cloudflare and network:</li>
<li>the hydrus client now has an experimental hook to the cloudscraper module, which is now an optional pip module for source users and included in all built releases. if a CF challenge page is downloaded, hydrus attempts to detect and solve it with cloudscraper and save the CF cookies back to the session before reattempting the request. all feedback on this working/breaking irl would be welcome. current expectation for this prototype is it can pass the basic 'wait five seconds' javascript challenge, and likely all but a handful of the more complicated captcha ones</li>
<li>if a CF challenge page is not solvable, the respective fail reason for that URL will be labelled appropriately about CloudFlare and have more technical information</li>
<li>.</li>
<li>the hydrus network engine now has the capability to remember recent serious network infrastructure errors (no connection, unsolvable cloudflare problem, etc..) on a per domain basis. if many serious errors have happened on a domain, new jobs will now wait until they are clear. this defaults to three or more such errors in the past ten minutes, and is configurable (and disableable) under options->connection. this will be built out to a flexible system in future, with per-domain options+status ui to see what's going on and actions to scrub delays</li>
<li>basically, if a server or your internet connection goes down, hydrus now throttles down to limit the damage</li>
<li>subscriptions now test if a domain is ok in order to decide whether they can start or continue file work, just like with bandwidth</li>
<li>serverside bandwidth alerts (429 or 509) are now classified as network infrastructure errors</li>
<li>I expect this system will need more tuning</li>
<li>.</li>
<li>the hydrus downloader system now recognises when an expected parseable document is actually an importable file. when this is true, the file is imported. this hopefully solves the situation where a site may deliver a post url or a file</li>
<li>.</li>
<li>the rest:</li>
<li>the windows build of hydrus is now in python 3.7.6, up from 3.6. this rolls in a host of small improvements, including to network stability and security (e.g. TLS 1.3), and possibly a couple of new bugs in more unusual hydrus systems</li>
<li>similarly, all the windows libraries are now their latest versions. opencv is now 4.2</li>
<li>greatly sped up several file searches that include no tags such as bare system:rating, most system file metadata predicates, or bare system:inbox, when the result size is much smaller than the total number of files in the file domain</li>
<li>thanks to some excellent work by a user, the Deviant Art downloader gets another pass--it can now get high res versions of images where they are available, and video, and flash, and pdf! the only proviso is that you need to be logged in to DA to get most content, otherwise you get 404. the current hydrus DA login script _seems_ to work ok</li>
<li>tag import options blacklists now test unnamespaced rules against namespaced tags. so if you blacklist 'metroid', a 'series:metroid' will be caught and the blacklist veto signal sent. this can be escaped with the 'advanced' exception panel, which now permits you to add 'redundant' rules</li>
<li>the edit tag filter panel now explains the blacklist rules explicitly and has a second 'test' green/red text to display test results for a tag import options blacklist, with the new sibling and namespace check</li>
<li>added some unit tests to test the new tag import options blacklist namespace rule</li>
<li>when 'default' tag import options are set, the edit panel now hides the per-service options, rather the the previous disable</li>
<li>the system tray icon now destroys itself when no longer needed, rather than hiding itself. it should now be more reliable in OSes that do not support system tray icon hide/show. if your OS still doesn't get rid of them, and you get a whole row of them, I recommend just leaving it always on</li>
<li>the system tray now has a tooltip with the main hydrus title and pause statuses</li>
<li>the timer that hides the mouse on the media viewer is now fired off when the window first opens (previously it would only initiate on the first mouse move over the window), so users who navigate mostly by keyboard should now see their cursors nicely hide on their own</li>
<li>added some semi-hacky import/export/duplicate buttons to edit shortcuts. I'll keep working on this, it'd be nice to have import/export for whole shortcut sets</li>
<li>added a semi-hacky duplicate button to the 'manage http headers' dialog</li>
<li>the 'clear' recent tag suggestions button is now wrapped in a yes/no dialog</li>
<li>a new checkbox under options->gui now lets you set it so when new cookies are sent from the API, or cookies are cleared, a popup message summarises the change. the popup dismisses itself after five seconds</li>
<li>the client api now also returns 'ext' on /get_files/file_metadata calls, just as a simpler alternative if the 'mime' is a pain</li>
<li>fixed a bug when petitioning tags through the client api, with or without reasons</li>
<li>fixed an error where subscriptions that somehow held invalid URLs would not be able to predict some bandwidth stuff, which would not allow the edit subs dialog to open</li>
<li>the string transformation dialog's step subdialog is now ok with example strings that are bytes. even then, this str/bytes dichotomy is an old artifact of python 2 and I will likely clean it up sometime so string transformers (and downloaders) only ever work utf-8 and hashes just work off utf-8 hex</li>
<li>added a BUGFIX checkbox to options->gui that tells the UI to use Qt file/directory picker dialogs, instead of the native OS one. users who have crashes on file selection are encouraged to try this out</li>
<li>updated running from source help with cloudscraper, a new pip masterline, and some windows venv info</li>
<li>the 'import with tags' button on 'import files' dialog gets another rename for new users, this time to 'add tags before the import &gt;&gt;'. it also gets a tooltip</li>
<li>handled an unusual rare error that could occur when switching out a media player inside a media viewer, perhaps during media viewer shutdown</li>
</ul>
<li><h3>version 392</h3></li>
<ul>
<li>db-level tag sibling cache:</li>

2
help/client_api.html
View File

@ -886,6 +886,7 @@
"hash" : "4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2",
"size" : 63405,
"mime" : "image/jpg",
"ext" : ".jpg",
"width" : 640,
"height" : 480,
"duration" : null,
@ -900,6 +901,7 @@
"hash" : "3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82",
"size" : 199713,
"mime" : "video/webm",
"ext" : ".webm",
"width" : 1920,
"height" : 1080,
"duration" : 4040,

10
help/running_from_source.html
View File

@ -27,6 +27,7 @@
<li>. venv/bin/activate</li>
</ul>
<p>That '. venv/bin/activate' line turns your venv on, and will be needed every time you run the client.pyw/server.py files. You can easily tuck it into a launch script.</p>
<p>On Windows, the path is venv&#92;Scripts&#92;activate, and the whole deal is done much easier in cmd than Powershell. If you get Powershell by default, just type 'cmd' to get an old fashioned command line. In cmd, the launch command is just 'venv&#92;scripts&#92;activate', no leading period.</p>
<p>After that, you can go nuts with pip. I think this will do for most systems:</p>
<ul>
<li>pip3 install beautifulsoup4 chardet html5lib lxml nose numpy opencv-python-headless six Pillow psutil PyOpenSSL PyYAML requests Send2Trash service_identity twisted</li>
@ -36,11 +37,16 @@
<ul>
<li>pip3 install qtpy PySide2</li>
</ul>
<p>I have had some stability trouble with the recent Qt 5.14, so you might like instead to do:</p>
<ul>
<li>pip3 install PySide2==5.13.2</li>
</ul>
<p>If your Linux package manager splits Qt into several pieces, you may not get QtCharts by default. This is optional but not required.</p>
<p>And optionally, you can add these packages:</p>
<ul>
<li>lz4 - for some memory compression in the client</li>
<li>pylzma - for importing rare ZWS swf files</li>
<li>cloudscraper - for attempting to solve CloudFlare check pages</li>
<li>pysocks - for socks4/socks5 proxy support (although you may want to try "requests[socks]" instead)</li>
<li>mock httmock pyinstaller - if you want to run test.py and make a build yourself</li>
</ul>
@ -48,6 +54,10 @@
<ul>
<li>pip3 install --upgrade libraryname</li>
</ul>
<p>Here is a masterline with everything for general use:</p>
<ul>
<li>pip3 install beautifulsoup4 chardet html5lib lxml nose numpy opencv-python-headless six Pillow psutil PyOpenSSL PyYAML requests Send2Trash service_identity twisted qtpy PySide2==5.13.2 lz4 pylzma cloudscraper pysocks</li>
</ul>
<p>For Windows, depending on which compiler you are using, pip can have problems building some modules like lz4 and lxml. <a href="http://www.lfd.uci.edu/~gohlke/pythonlibs/">This page</a> has a lot of prebuilt binaries--I have found it very helpful many times. You may want to update python's sqlite3.dll as well--you can get it <a href="https://www.sqlite.org/download.html">here</a>, and just drop it in C:\Python37\DLLs or wherever you have python installed. I have a fair bit of experience with Windows python, so send me a mail if you need help.</a>
<p>If you don't have ffmpeg in your PATH and you want to import videos, you will need to put a static <a href="https://ffmpeg.org/">FFMPEG</a> executable in the install_dir/bin directory. Have a look at how I do it in the extractable compiled releases if you can't figure it out. On Windows, you can copy the exe from one of those releases, or just download the latest static build right from the FFMPEG site.</a>
<p>Once you have everything set up, client.pyw and server.py should look for and run off client.db and server.db just like the executables. They will look in the 'db' directory by default, or anywhere you point them with the "-d" parameter, again just like the executables.</p>

55
hydrus/ClientDB.py
View File

@ -5852,7 +5852,26 @@ class DB( HydrusDB.HydrusDB ):
files_info_predicates.insert( 0, 'service_id = ' + str( file_service_id ) )
query_hash_ids = intersection_update_qhi( query_hash_ids, self._STS( self._c.execute( 'SELECT hash_id FROM current_files NATURAL JOIN files_info WHERE ' + ' AND '.join( files_info_predicates ) + ';' ) ) )
if query_hash_ids is None:
query_hash_ids = intersection_update_qhi( query_hash_ids, self._STS( self._c.execute( 'SELECT hash_id FROM current_files NATURAL JOIN files_info WHERE {};'.format( ' AND '.join( files_info_predicates ) ) ) ) )
else:
if is_inbox and len( query_hash_ids ) == len( self._inbox_hash_ids ):
query_hash_ids = intersection_update_qhi( query_hash_ids, self._STS( self._c.execute( 'SELECT hash_id FROM current_files NATURAL JOIN files_info NATURAL JOIN {} WHERE {};'.format( 'file_inbox', ' AND '.join( files_info_predicates ) ) ) ) )
else:
with HydrusDB.TemporaryIntegerTable( self._c, query_hash_ids, 'hash_id' ) as temp_table_name:
self._AnalyzeTempTable( temp_table_name )
query_hash_ids = intersection_update_qhi( query_hash_ids, self._STS( self._c.execute( 'SELECT hash_id FROM current_files NATURAL JOIN files_info NATURAL JOIN {} WHERE {};'.format( temp_table_name, ' AND '.join( files_info_predicates ) ) ) ) )
have_cross_referenced_file_service = True
done_files_info_predicates = True
@ -14312,6 +14331,40 @@ class DB( HydrusDB.HydrusDB ):
if version == 392:
try:
domain_manager = self._GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER )
domain_manager.Initialise()
#
domain_manager.OverwriteDefaultURLClasses( [ 'deviant art file page extended_fetch api', 'deviant art file page', 'deviant art flash sandbox page' ] )
#
domain_manager.OverwriteDefaultParsers( [ 'deviant art flash sandbox page parser', 'deviant art file extended_fetch parser' ] )
#
domain_manager.TryToLinkURLClassesAndParsers()
#
self._SetJSONDump( domain_manager )
except Exception as e:
HydrusData.PrintException( e )
message = 'Trying to update some parsers failed! Please let hydrus dev know!'
self.pub_initial_message( message )
self._controller.pub( 'splash_set_title_text', 'updated db to v{}'.format( HydrusData.ToHumanInt( version + 1 ) ) )
self._c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )

25
hydrus/ClientGUI.py
View File

@ -565,6 +565,10 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ):
library_versions.append( ( 'html5lib present: ', str( ClientParsing.HTML5LIB_IS_OK ) ) )
library_versions.append( ( 'lxml present: ', str( ClientParsing.LXML_IS_OK ) ) )
from . import ClientNetworkingJobs
library_versions.append( ( 'cloudscraper present: ', str( ClientNetworkingJobs.CLOUDSCRAPER_OK ) ) )
library_versions.append( ( 'lz4 present: ', str( ClientRendering.LZ4_OK ) ) )
library_versions.append( ( 'install dir', HC.BASE_DIR ) )
library_versions.append( ( 'db dir', HG.client_controller.db_dir ) )
@ -2453,7 +2457,7 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ):
def _ManageNetworkHeaders( self ):
title = 'manage network headers'
title = 'manage http headers'
with ClientGUITopLevelWindows.DialogEdit( self, title ) as dlg:
@ -3740,20 +3744,21 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
self._system_tray_icon.show()
self._system_tray_icon.SetShouldAlwaysShow( always_show_system_tray_icon )
self._system_tray_icon.SetUIIsCurrentlyShown( not self._currently_minimised_to_system_tray )
self._system_tray_icon.SetNetworkTrafficPaused( new_options.GetBoolean( 'pause_all_new_network_traffic' ) )
self._system_tray_icon.SetSubscriptionsPaused( HC.options[ 'pause_subs_sync' ] )
else:
if self._have_system_tray_icon:
self._system_tray_icon.hide()
self._system_tray_icon.deleteLater()
self._system_tray_icon = None
self._have_system_tray_icon = False
if self._have_system_tray_icon:
self._system_tray_icon.SetShouldAlwaysShow( always_show_system_tray_icon )
self._system_tray_icon.SetUIIsCurrentlyShown( not self._currently_minimised_to_system_tray )
self._system_tray_icon.SetNetworkTrafficPaused( self._controller.new_options.GetBoolean( 'pause_all_new_network_traffic' ) )
self._system_tray_icon.SetSubscriptionsPaused( HC.options[ 'pause_subs_sync' ] )

2
hydrus/ClientGUICanvas.py
View File

@ -2166,6 +2166,8 @@ class CanvasWithHovers( CanvasWithDetails ):
self._widget_event_filter.EVT_MOTION( self.EventMouseMove )
self._InitiateCursorHideWait()
HG.client_controller.sub( self, 'CloseFromHover', 'canvas_close' )
HG.client_controller.sub( self, 'FullscreenSwitch', 'canvas_fullscreen_switch' )

39
hydrus/ClientGUICanvasMedia.py
View File

@ -915,23 +915,16 @@ class MediaContainer( QW.QWidget ):
if media_window is not None:
if isinstance( media_window, ( Animation, StaticImage ) ):
media_window.launchMediaViewer.disconnect( self.launchMediaViewer )
if isinstance( media_window, ( Animation, StaticImage, ClientGUIMPV.mpvWidget ) ):
media_window.ClearMedia()
media_window.hide()
elif isinstance( media_window, ClientGUIMPV.mpvWidget ):
media_window.launchMediaViewer.disconnect( self.launchMediaViewer )
media_window.ClearMedia()
media_window.hide()
HG.client_controller.gui.ReleaseMPVWidget( media_window )
if isinstance( media_window, ClientGUIMPV.mpvWidget ):
HG.client_controller.gui.ReleaseMPVWidget( media_window )
else:
@ -1041,11 +1034,6 @@ class MediaContainer( QW.QWidget ):
self._media_window.installEventFilter( self._drag_click_reporting_filter )
if old_media_window is not None:
old_media_window.removeEventFilter( self._drag_click_reporting_filter )
launch_media_viewer_classes = ( Animation, ClientGUIMPV.mpvWidget, StaticImage )
if isinstance( self._media_window, launch_media_viewer_classes ):
@ -1053,6 +1041,23 @@ class MediaContainer( QW.QWidget ):
self._media_window.launchMediaViewer.connect( self.launchMediaViewer )
if old_media_window is not None:
old_media_window.removeEventFilter( self._drag_click_reporting_filter )
if isinstance( old_media_window, launch_media_viewer_classes ):
try:
old_media_window.launchMediaViewer.disconnect( self.launchMediaViewer )
except RuntimeError:
pass # lmao, weird 'Failed to disconnect signal launchMediaViewer()' error I couldn't figure out, I guess some out-of-order deleteLater gubbins
self._media_window

19
hydrus/ClientGUIControls.py
View File

@ -679,7 +679,18 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
self._example_string.setMinimumWidth( min_width )
self._example_string.setText( example_text )
self._example_text = example_text
if isinstance( self._example_text, bytes ):
self._example_string.setText( repr( self._example_text ) )
else:
self._example_string.setText( self._example_text )
self._example_transformation = QW.QLineEdit( self )
@ -938,11 +949,9 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
transformations = [ self.GetValue() ]
example_string = self._example_string.text()
string_converter = ClientParsing.StringConverter( transformations, self._example_text )
string_converter = ClientParsing.StringConverter( transformations, example_string )
example_transformation = string_converter.Convert( example_string )
example_transformation = string_converter.Convert( self._example_text )
try:

32
hydrus/ClientGUIListCtrl.py
View File

@ -600,8 +600,8 @@ class BetterListCtrl( QW.QTreeWidget ):
self._data_to_indices[ new_data ] = data_index
self._UpdateRow( data_index, display_tuple )
class BetterListCtrlPanel( QW.QWidget ):
def __init__( self, parent ):
@ -616,6 +616,7 @@ class BetterListCtrlPanel( QW.QWidget ):
self._permitted_object_types = []
self._import_add_callable = lambda x: None
self._custom_get_callable = None
self._button_infos = []
@ -760,11 +761,18 @@ class BetterListCtrlPanel( QW.QWidget ):
def _GetExportObject( self ):
to_export = HydrusSerialisable.SerialisableList()
for obj in self._listctrl.GetData( only_selected = True ):
if self._custom_get_callable is None:
to_export.append( obj )
to_export = HydrusSerialisable.SerialisableList()
for obj in self._listctrl.GetData( only_selected = True ):
to_export.append( obj )
else:
to_export = [ self._custom_get_callable() ]
if len( to_export ) == 0:
@ -965,21 +973,25 @@ class BetterListCtrlPanel( QW.QWidget ):
self.AddButton( 'delete', self._listctrl.ProcessDeleteAction, enabled_check_func = enabled_check_func, enabled_only_on_selection = enabled_only_on_selection )
def AddImportExportButtons( self, permitted_object_types, import_add_callable ):
def AddImportExportButtons( self, permitted_object_types, import_add_callable, custom_get_callable = None ):
self._permitted_object_types = permitted_object_types
self._import_add_callable = import_add_callable
self._custom_get_callable = custom_get_callable
export_menu_items = []
export_menu_items.append( ( 'normal', 'to clipboard', 'Serialise the selected data and put it on your clipboard.', self._ExportToClipboard ) )
export_menu_items.append( ( 'normal', 'to png', 'Serialise the selected data and encode it to an image file you can easily share with other hydrus users.', self._ExportToPng ) )
all_objs_are_named = False not in ( issubclass( o, HydrusSerialisable.SerialisableBaseNamed ) for o in self._permitted_object_types )
if all_objs_are_named:
if self._custom_get_callable is None:
export_menu_items.append( ( 'normal', 'to pngs', 'Serialise the selected data and encode it to multiple image files you can easily share with other hydrus users.', self._ExportToPngs ) )
all_objs_are_named = False not in ( issubclass( o, HydrusSerialisable.SerialisableBaseNamed ) for o in self._permitted_object_types )
if all_objs_are_named:
export_menu_items.append( ( 'normal', 'to pngs', 'Serialise the selected data and encode it to multiple image files you can easily share with other hydrus users.', self._ExportToPngs ) )
import_menu_items = []

101
hydrus/ClientGUIScrolledPanelsEdit.py
View File

@ -2957,6 +2957,7 @@ class EditNetworkContextCustomHeadersPanel( ClientGUIScrolledPanels.EditPanel ):
self._list_ctrl_panel.AddButton( 'add', self._Add )
self._list_ctrl_panel.AddButton( 'edit', self._Edit, enabled_only_on_selection = True )
self._list_ctrl_panel.AddDeleteButton()
self._list_ctrl_panel.AddButton( 'duplicate', self._Duplicate, enabled_only_on_selection = True )
self._list_ctrl.Sort( 0 )
@ -3025,6 +3026,22 @@ class EditNetworkContextCustomHeadersPanel( ClientGUIScrolledPanels.EditPanel ):
return ( display_tuple, sort_tuple )
def _Duplicate( self ):
existing_keys = { key for ( network_context, ( key, value ), approved, reason ) in self._list_ctrl.GetData() }
datas = self._list_ctrl.GetData( only_selected = True )
for ( network_context, ( key, value ), approved, reason ) in datas:
key = HydrusData.GetNonDupeName( key, existing_keys )
existing_keys.add( key )
self._list_ctrl.AddDatas( [ ( network_context, ( key, value ), approved, reason ) ] )
def _Edit( self ):
for data in self._list_ctrl.GetData( only_selected = True ):
@ -3830,18 +3847,26 @@ But if 2 is--and is also perhaps accompanied by many 'could not parse' errors--t
file_velocity = checker_options.GetRawCurrentVelocity( query.GetFileSeedCache(), last_check_time )
pretty_file_velocity = checker_options.GetPrettyCurrentVelocity( query.GetFileSeedCache(), last_check_time, no_prefix = True )
estimate = query.GetBandwidthWaitingEstimate( self._original_subscription.GetName() )
if estimate == 0:
try:
pretty_delay = ''
estimate = query.GetBandwidthWaitingEstimate( self._original_subscription.GetName() )
if estimate == 0:
pretty_delay = ''
delay = 0
else:
pretty_delay = 'bandwidth: ' + HydrusData.TimeDeltaToPrettyTimeDelta( estimate )
delay = estimate
except:
pretty_delay = 'could not determine bandwidth--there may be a problem with some of the urls in this query'
delay = 0
else:
pretty_delay = 'bandwidth: ' + HydrusData.TimeDeltaToPrettyTimeDelta( estimate )
delay = estimate
( file_status, simple_status, ( num_done, num_total ) ) = file_seed_cache.GetStatus()
@ -4641,31 +4666,39 @@ class EditSubscriptionsPanel( ClientGUIScrolledPanels.EditPanel ):
if HydrusData.TimeHasPassed( no_work_until ):
( min_estimate, max_estimate ) = subscription.GetBandwidthWaitingEstimateMinMax()
if max_estimate == 0: # don't seem to be any delays of any kind
try:
pretty_delay = ''
delay = 0
( min_estimate, max_estimate ) = subscription.GetBandwidthWaitingEstimateMinMax()
elif min_estimate == 0: # some are good to go, but there are delays
pretty_delay = 'bandwidth: some ok, some up to ' + HydrusData.TimeDeltaToPrettyTimeDelta( max_estimate )
delay = max_estimate
else:
if min_estimate == max_estimate: # probably just one query, and it is delayed
if max_estimate == 0: # don't seem to be any delays of any kind
pretty_delay = 'bandwidth: up to ' + HydrusData.TimeDeltaToPrettyTimeDelta( max_estimate )
pretty_delay = ''
delay = 0
elif min_estimate == 0: # some are good to go, but there are delays
pretty_delay = 'bandwidth: some ok, some up to ' + HydrusData.TimeDeltaToPrettyTimeDelta( max_estimate )
delay = max_estimate
else:
pretty_delay = 'bandwidth: from ' + HydrusData.TimeDeltaToPrettyTimeDelta( min_estimate ) + ' to ' + HydrusData.TimeDeltaToPrettyTimeDelta( max_estimate )
delay = max_estimate
if min_estimate == max_estimate: # probably just one query, and it is delayed
pretty_delay = 'bandwidth: up to ' + HydrusData.TimeDeltaToPrettyTimeDelta( max_estimate )
delay = max_estimate
else:
pretty_delay = 'bandwidth: from ' + HydrusData.TimeDeltaToPrettyTimeDelta( min_estimate ) + ' to ' + HydrusData.TimeDeltaToPrettyTimeDelta( max_estimate )
delay = max_estimate
except:
pretty_delay = 'could not determine bandwidth, there may be an error with the sub or its urls'
delay = 0
else:
@ -5338,9 +5371,13 @@ class EditTagImportOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
tag_blacklist = tag_import_options.GetTagBlacklist()
message = 'Any tag that this filter _excludes_ will be considered a blacklisted tag and will stop the file importing.'
message = 'Any tag that this filter _excludes_ will be considered a blacklisted tag and will stop the file importing. So if you only want to stop \'scat\' or \'gore\', just add them to the simple blacklist and hit ok.'
message += os.linesep * 2
message += 'So if you only want to stop \'scat\' or \'gore\', just add them to the simple blacklist and hit ok. It is worth doing a small test, just to make sure it is all set up how you want.'
message += 'This system tests the tags that are parsed from the site, as hydrus would end up getting them. Siblings of all the tags will also be tested. If you do not have excellent siblings, it is worth adding multiple versions of your tag, just to catch different sites terms. Add \'gore\', \'guro\', \'violence\', etc...'
message += os.linesep * 2
message += 'Additionally, for blacklists, unnamespaced rules will apply to namespaced tags. \'metroid\' in the blacklist will catch \'series:metroid\' as parsed from a site.'
message += os.linesep * 2
message += 'It is worth doing a small test here, just to make sure it is all set up how you want.'
self._tag_filter_button = ClientGUITags.TagFilterButton( downloader_options_panel, message, tag_blacklist, is_blacklist = True )
@ -5413,6 +5450,7 @@ class EditTagImportOptionsPanel( ClientGUIScrolledPanels.EditPanel ):
QP.AddToLayout( vbox, help_button, CC.FLAGS_LONE_BUTTON )
QP.AddToLayout( vbox, default_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._specific_options_panel, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
QP.AddToLayout( vbox, QW.QWidget( self ), CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
self.widget().setLayout( vbox )
@ -5528,7 +5566,12 @@ Please note that once you know what tags you like, you can (and should) set up t
show_specific_options = not is_default
self._specific_options_panel.setEnabled( show_specific_options )
self._specific_options_panel.setVisible( show_specific_options )
if not show_specific_options:
self.window().adjustSize()
def GetValue( self ):
@ -5544,7 +5587,7 @@ Please note that once you know what tags you like, you can (and should) set up t
fetch_tags_even_if_url_recognised_and_file_already_in_db = self._fetch_tags_even_if_url_recognised_and_file_already_in_db.isChecked()
fetch_tags_even_if_hash_recognised_and_file_already_in_db = self._fetch_tags_even_if_hash_recognised_and_file_already_in_db.isChecked()
service_keys_to_service_tag_import_options = {service_key : panel.GetValue() for (service_key, panel) in list( self._service_keys_to_service_tag_import_options_panels.items() )}
service_keys_to_service_tag_import_options = { service_key : panel.GetValue() for ( service_key, panel ) in list( self._service_keys_to_service_tag_import_options_panels.items() ) }
tag_blacklist = self._tag_filter_button.GetValue()

28
hydrus/ClientGUIScrolledPanelsManagement.py
View File

@ -1753,6 +1753,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._serverside_bandwidth_wait_time = QP.MakeQSpinBox( general, min = error_wait_time_min, max = error_wait_time_max )
self._serverside_bandwidth_wait_time.setToolTip( 'If a server returns a failure status code indicating it is short on bandwidth, the network job will wait increasing multiples of this base time before retrying.' )
self._domain_network_infrastructure_error_velocity = ClientGUITime.VelocityCtrl( general, 0, 100, 30, hours = True, minutes = True, seconds = True, per_phrase = 'within', unit = 'errors' )
self._max_network_jobs = QP.MakeQSpinBox( general, min = 1, max = max_network_jobs_max )
self._max_network_jobs_per_domain = QP.MakeQSpinBox( general, min = 1, max = max_network_jobs_per_domain_max )
@ -1774,6 +1776,11 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._connection_error_wait_time.setValue( self._new_options.GetInteger( 'connection_error_wait_time' ) )
self._serverside_bandwidth_wait_time.setValue( self._new_options.GetInteger( 'serverside_bandwidth_wait_time' ) )
number = self._new_options.GetInteger( 'domain_network_infrastructure_error_number' )
time_delta = self._new_options.GetInteger( 'domain_network_infrastructure_error_time_delta' )
self._domain_network_infrastructure_error_velocity.SetValue( ( number, time_delta ) )
self._max_network_jobs.setValue( self._new_options.GetInteger( 'max_network_jobs' ) )
self._max_network_jobs_per_domain.setValue( self._new_options.GetInteger( 'max_network_jobs_per_domain' ) )
@ -1796,6 +1803,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
rows.append( ( 'network timeout (seconds): ', self._network_timeout ) )
rows.append( ( 'connection error retry wait (seconds): ', self._connection_error_wait_time ) )
rows.append( ( 'serverside bandwidth retry wait (seconds): ', self._serverside_bandwidth_wait_time ) )
rows.append( ( 'Halt new jobs as long as this many network infrastructure errors on their domain (0 for never wait): ', self._domain_network_infrastructure_error_velocity ) )
rows.append( ( 'max number of simultaneous active network jobs: ', self._max_network_jobs ) )
rows.append( ( 'max number of simultaneous active network jobs per domain: ', self._max_network_jobs_per_domain ) )
rows.append( ( 'BUGFIX: verify regular https traffic:', self._verify_regular_https ) )
@ -1857,6 +1865,11 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._new_options.SetInteger( 'max_network_jobs', self._max_network_jobs.value() )
self._new_options.SetInteger( 'max_network_jobs_per_domain', self._max_network_jobs_per_domain.value() )
( number, time_delta ) = self._domain_network_infrastructure_error_velocity.GetValue()
self._new_options.SetInteger( 'domain_network_infrastructure_error_number', number )
self._new_options.SetInteger( 'domain_network_infrastructure_error_time_delta', time_delta )
class _DownloadingPanel( QW.QWidget ):
@ -2171,6 +2184,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._new_options = new_options
self._always_show_system_everything = QW.QCheckBox( 'show system:everything even if total files is over 10,000', self )
self._always_show_system_everything.setToolTip( 'After users get some experience with the program and a larger collection, they tend to have less use for system:everything.' )
self._always_show_system_everything.setChecked( self._new_options.GetBoolean( 'always_show_system_everything' ) )
@ -2646,6 +2660,12 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._hide_message_manager_on_gui_deactive = QW.QCheckBox( self )
self._hide_message_manager_on_gui_deactive.setToolTip( 'If your message manager stays up after you minimise the program to the system tray using a custom window manager, try this out! It hides the popup messages as soon as the main gui loses focus.' )
self._notify_client_api_cookies = QW.QCheckBox( self )
self._notify_client_api_cookies.setToolTip( 'This will make a short-lived popup message every time you get new cookie information over the Client API.' )
self._use_qt_file_dialogs = QW.QCheckBox( self )
self._use_qt_file_dialogs.setToolTip( 'If you get crashes opening file/directory dialogs, try this.' )
frame_locations_panel = ClientGUICommon.StaticBox( self, 'frame locations' )
self._frame_locations = ClientGUIListCtrl.BetterListCtrl( frame_locations_panel, 'frame_locations', 15, 20, [ ( 'name', -1 ), ( 'remember size', 12 ), ( 'remember position', 12 ), ( 'last size', 12 ), ( 'last position', 12 ), ( 'default gravity', 12 ), ( 'default position', 12 ), ( 'maximised', 12 ), ( 'fullscreen', 12 ) ], data_to_tuples_func = lambda x: (self._GetPrettyFrameLocationInfo( x ), self._GetPrettyFrameLocationInfo( x )), activation_callback = self.EditFrameLocations )
@ -2679,6 +2699,10 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._hide_message_manager_on_gui_iconise.setChecked( self._new_options.GetBoolean( 'hide_message_manager_on_gui_iconise' ) )
self._hide_message_manager_on_gui_deactive.setChecked( self._new_options.GetBoolean( 'hide_message_manager_on_gui_deactive' ) )
self._notify_client_api_cookies.setChecked( self._new_options.GetBoolean( 'notify_client_api_cookies' ) )
self._use_qt_file_dialogs.setChecked( self._new_options.GetBoolean( 'use_qt_file_dialogs' ) )
for ( name, info ) in self._new_options.GetFrameLocations():
listctrl_list = QP.ListsToTuples( [ name ] + list( info ) )
@ -2699,11 +2723,13 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
rows.append( ( 'Autocomplete results float in other windows: ', self._autocomplete_float_frames ) )
rows.append( ( 'Hide the preview window: ', self._hide_preview ) )
rows.append( ( 'Approximate max width of popup messages (in characters): ', self._popup_message_character_width ) )
rows.append( ( 'Make a short-lived popup on cookie updates through the Client API: ', self._notify_client_api_cookies ) )
rows.append( ( 'BUGFIX: Force this width as the minimum width for all popup messages: ', self._popup_message_force_min_width ) )
rows.append( ( 'BUGFIX: Discord file drag-and-drop fix (works for <=25, <200MB file DnDs): ', self._discord_dnd_fix ) )
rows.append( ( 'EXPERIMENTAL BUGFIX: Secret discord file drag-and-drop fix: ', self._secret_discord_dnd_fix ) )
rows.append( ( 'BUGFIX: Hide the popup message manager when the main gui is minimised: ', self._hide_message_manager_on_gui_iconise ) )
rows.append( ( 'BUGFIX: Hide the popup message manager when the main gui loses focus: ', self._hide_message_manager_on_gui_deactive ) )
rows.append( ( 'ANTI-CRASH BUGFIX: Use Qt file/directory selection dialogs, rather than OS native: ', self._use_qt_file_dialogs ) )
gridbox = ClientGUICommon.WrapInGrid( self, rows )
@ -2778,10 +2804,12 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
HG.client_controller.pub( 'main_gui_title', title )
self._new_options.SetBoolean( 'notify_client_api_cookies', self._notify_client_api_cookies.isChecked() )
self._new_options.SetBoolean( 'discord_dnd_fix', self._discord_dnd_fix.isChecked() )
self._new_options.SetBoolean( 'secret_discord_dnd_fix', self._secret_discord_dnd_fix.isChecked() )
self._new_options.SetBoolean( 'hide_message_manager_on_gui_iconise', self._hide_message_manager_on_gui_iconise.isChecked() )
self._new_options.SetBoolean( 'hide_message_manager_on_gui_deactive', self._hide_message_manager_on_gui_deactive.isChecked() )
self._new_options.SetBoolean( 'use_qt_file_dialogs', self._use_qt_file_dialogs.isChecked() )
for listctrl_list in self._frame_locations.GetData():

4
hydrus/ClientGUIScrolledPanelsReview.py
View File

@ -2992,9 +2992,11 @@ class ReviewLocalFileImports( ClientGUIScrolledPanels.ReviewPanel ):
self._add_button = ClientGUICommon.BetterButton( self, 'import now', self._DoImport )
self._add_button.setObjectName( 'HydrusAccept' )
self._tag_button = ClientGUICommon.BetterButton( self, 'import with tags', self._AddTags )
self._tag_button = ClientGUICommon.BetterButton( self, 'add tags before the import >>', self._AddTags )
self._tag_button.setObjectName( 'HydrusAccept' )
self._tag_button.setToolTip( 'You can add specific tags to these files, import from sidecar files, or generate them based on filename. Don\'t be afraid to experiment!' )
gauge_sizer = QP.HBoxLayout()
QP.AddToLayout( gauge_sizer, self._progress, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )

33
hydrus/ClientGUIShortcutControls.py
View File

@ -612,13 +612,19 @@ class EditShortcutAndCommandPanel( ClientGUIScrolledPanels.EditPanel ):
class EditShortcutSetPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent, shortcuts ):
def __init__( self, parent, shortcuts: ClientGUIShortcuts.ShortcutSet ):
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
self._name = QW.QLineEdit( self )
self._shortcuts = ClientGUIListCtrl.BetterListCtrl( self, 'shortcuts', 20, 20, [ ( 'shortcut', 20 ), ( 'command', -1 ) ], data_to_tuples_func = self._ConvertSortTupleToPrettyTuple, delete_key_callback = self.RemoveShortcuts, activation_callback = self.EditShortcuts )
self._shortcuts_panel = ClientGUIListCtrl.BetterListCtrlPanel( self )
self._shortcuts = ClientGUIListCtrl.BetterListCtrl( self._shortcuts_panel, 'shortcuts', 20, 20, [ ( 'shortcut', 20 ), ( 'command', -1 ) ], data_to_tuples_func = self._ConvertSortTupleToPrettyTuple, delete_key_callback = self.RemoveShortcuts, activation_callback = self.EditShortcuts )
self._shortcuts_panel.SetListCtrl( self._shortcuts )
self._shortcuts_panel.AddImportExportButtons( ( ClientGUIShortcuts.ShortcutSet, ), self._AddShortcutSet, custom_get_callable = self._GetSelectedShortcutSet )
self._shortcuts.setMinimumSize( QC.QSize( 360, 480 ) )
@ -645,8 +651,8 @@ class EditShortcutSetPanel( ClientGUIScrolledPanels.EditPanel ):
self._name.setEnabled( False )
self._shortcuts.AddDatas( shortcuts )
self._shortcuts.Sort( 1 )
@ -673,7 +679,7 @@ class EditShortcutSetPanel( ClientGUIScrolledPanels.EditPanel ):
QP.AddToLayout( vbox, description, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._shortcuts, CC.FLAGS_EXPAND_BOTH_WAYS )
QP.AddToLayout( vbox, self._shortcuts_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
QP.AddToLayout( vbox, action_buttons, CC.FLAGS_BUTTON_SIZER )
self.widget().setLayout( vbox )
@ -689,6 +695,25 @@ class EditShortcutSetPanel( ClientGUIScrolledPanels.EditPanel ):
return ( display_tuple, sort_tuple )
def _AddShortcutSet( self, shortcut_set: ClientGUIShortcuts.ShortcutSet ):
self._shortcuts.AddDatas( shortcut_set )
def _GetSelectedShortcutSet( self ):
name = self._name.text()
shortcut_set = ClientGUIShortcuts.ShortcutSet( name )
for ( shortcut, command ) in self._shortcuts.GetData( only_selected = True ):
shortcut_set.SetCommand( shortcut, command )
return shortcut_set
def AddShortcut( self ):
shortcut = ClientGUIShortcuts.Shortcut()

33
hydrus/ClientGUISystemTray.py
View File

@ -78,6 +78,8 @@ class ClientSystemTrayIcon( QW.QSystemTrayIcon ):
ClientGUIMenus.DestroyMenu( parent_widget, old_menu )
self._UpdateTooltip()
def _UpdateNetworkTrafficMenuItemLabel( self ):
@ -126,6 +128,33 @@ class ClientSystemTrayIcon( QW.QSystemTrayIcon ):
def _UpdateTooltip( self ):
title = HG.client_controller.new_options.GetString( 'main_gui_title' )
if title is None or title == '':
title = 'hydrus client'
tooltip = title
if self._network_traffic_paused:
tooltip = '{} - network traffic paused'.format( tooltip )
if self._subscriptions_paused:
tooltip = '{} - subscriptions paused'.format( tooltip )
if self.toolTip != tooltip:
self.setToolTip( tooltip )
def _WasActivated( self, activation_reason ):
if activation_reason in ( QW.QSystemTrayIcon.Unknown, QW.QSystemTrayIcon.Trigger ):
@ -162,6 +191,8 @@ class ClientSystemTrayIcon( QW.QSystemTrayIcon ):
self._UpdateNetworkTrafficMenuItemLabel()
self._UpdateTooltip()
def SetSubscriptionsPaused( self, subscriptions_paused: bool ):
@ -172,6 +203,8 @@ class ClientSystemTrayIcon( QW.QSystemTrayIcon ):
self._UpdateSubscriptionsMenuItemLabel()
self._UpdateTooltip()
def SetUIIsCurrentlyShown( self, ui_is_currently_shown: bool ):

11
hydrus/ClientGUITagSuggestions.py
View File

@ -171,9 +171,16 @@ class RecentTagsPanel( QW.QWidget ):
def EventClear( self ):
HG.client_controller.Write( 'push_recent_tags', self._service_key, None )
from . import ClientGUIDialogsQuick
self._RefreshRecentTags()
result = ClientGUIDialogsQuick.GetYesNo( self, 'Clear recent tags?' )
if result == QW.QDialog.Accepted:
HG.client_controller.Write( 'push_recent_tags', self._service_key, None )
self._RefreshRecentTags()
def RefreshRecentTags( self ):

60
hydrus/ClientGUITags.py
View File

@ -153,6 +153,7 @@ class EditTagDisplayManagerPanel( ClientGUIScrolledPanels.EditPanel ):
class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
TEST_RESULT_DEFAULT = 'Enter a tag here to test if it passes the current filter:'
TEST_RESULT_BLACKLIST_DEFAULT = 'Enter a tag here to test if it passes the current filter in a tag import options blacklist (siblings tested, unnamespaced rules match namespaced tags):'
def __init__( self, parent, tag_filter, prefer_blacklist = False, namespaces = None, message = None ):
@ -216,6 +217,13 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
self._test_result_st = ClientGUICommon.BetterStaticText( self, self.TEST_RESULT_DEFAULT )
self._test_result_st.setAlignment( QC.Qt.AlignVCenter | QC.Qt.AlignRight )
self._test_result_st.setWordWrap( True )
self._test_result_blacklist_st = ClientGUICommon.BetterStaticText( self, self.TEST_RESULT_BLACKLIST_DEFAULT )
self._test_result_blacklist_st.setAlignment( QC.Qt.AlignVCenter | QC.Qt.AlignRight )
self._test_result_blacklist_st.setWordWrap( True )
self._test_input = QW.QPlainTextEdit( self )
#
@ -242,9 +250,14 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
QP.AddToLayout( vbox, self._redundant_st, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._current_filter_st, CC.FLAGS_EXPAND_PERPENDICULAR )
test_text_vbox = QP.VBoxLayout()
QP.AddToLayout( test_text_vbox, self._test_result_st, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( test_text_vbox, self._test_result_blacklist_st, CC.FLAGS_EXPAND_PERPENDICULAR )
hbox = QP.HBoxLayout()
QP.AddToLayout( hbox, self._test_result_st, CC.FLAGS_VCENTER_EXPAND_DEPTH_ONLY )
QP.AddToLayout( hbox, test_text_vbox, CC.FLAGS_VCENTER_EXPAND_DEPTH_ONLY )
QP.AddToLayout( hbox, self._test_input, CC.FLAGS_VCENTER_EXPAND_DEPTH_ONLY )
QP.AddToLayout( vbox, hbox, CC.FLAGS_EXPAND_PERPENDICULAR )
@ -281,10 +294,8 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
self._ShowRedundantError( ClientTags.ConvertTagSliceToString( tag_slice ) + ' is already blocked by a broader rule!' )
else:
self._advanced_blacklist.AddTags( ( tag_slice, ) )
self._advanced_blacklist.AddTags( ( tag_slice, ) )
self._UpdateStatus()
@ -319,15 +330,13 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
# if it is still blocked after that, it needs whitelisting explicitly
if self._CurrentlyBlocked( tag_slice ):
self._advanced_whitelist.AddTags( ( tag_slice, ) )
elif tag_slice not in ( '', ':' ):
if not self._CurrentlyBlocked( tag_slice ) and tag_slice not in ( '', ':' ):
self._ShowRedundantError( ClientTags.ConvertTagSliceToString( tag_slice ) + ' is already permitted by a broader rule!' )
self._advanced_whitelist.AddTags( ( tag_slice, ) )
self._UpdateStatus()
@ -1061,9 +1070,8 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
if test_input == '':
text = self.TEST_RESULT_DEFAULT
colour = QP.GetSystemColour( QG.QPalette.WindowText )
normal_text = self.TEST_RESULT_DEFAULT
blacklist_text = self.TEST_RESULT_BLACKLIST_DEFAULT
else:
@ -1071,21 +1079,41 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
if tag_filter.TagOK( test_input ):
text = 'tag passes!'
normal_text = 'tag passes!'
self._test_result_st.setObjectName( 'HydrusValid' )
else:
text = 'tag blocked!'
normal_text = 'tag blocked!'
self._test_result_st.setObjectName( 'HydrusInvalid' )
sibling_tags = HG.client_controller.tag_siblings_manager.GetAllSiblings( CC.COMBINED_TAG_SERVICE_KEY, test_input )
passes = False not in ( tag_filter.TagOK( sibling_tag, apply_unnamespaced_rules_to_namespaced_tags = True ) for sibling_tag in sibling_tags )
if passes:
blacklist_text = 'in a tag import options blacklist, tag passes!'
self._test_result_blacklist_st.setObjectName( 'HydrusValid' )
else:
blacklist_text = 'in a tag import options blacklist, tag blocked!'
self._test_result_blacklist_st.setObjectName( 'HydrusInvalid' )
self._test_result_st.setText( text )
self._test_result_st.setText( normal_text )
self._test_result_st.style().polish( self._test_result_st )
self._test_result_blacklist_st.setText( blacklist_text )
self._test_result_blacklist_st.style().polish( self._test_result_blacklist_st )
def EventSimpleBlacklistNamespaceCheck( self, index ):

36
hydrus/ClientImportFileSeeds.py
View File

@ -1266,7 +1266,41 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
if len( all_parse_results ) == 0:
raise HydrusExceptions.VetoException( 'The parser found nothing in the document!' )
it_was_a_real_file = False
( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
try:
with open( temp_path, 'wb' ) as f:
f.write( network_job.GetContentBytes() )
mime = HydrusFileHandling.GetMime( temp_path )
if mime in HC.ALLOWED_MIMES:
it_was_a_real_file = True
status_hook( 'page was actually a file, trying to import' )
self.Import( temp_path, file_import_options, status_hook = status_hook )
except:
pass # in this special occasion, we will swallow the error
finally:
HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
if not it_was_a_real_file:
raise HydrusExceptions.VetoException( 'The parser found nothing in the document, nor did it seem to be an importable file!' )
elif len( all_parse_results ) > 1:

2
hydrus/ClientImportOptions.py
View File

@ -1113,7 +1113,7 @@ class TagImportOptions( HydrusSerialisable.SerialisableBase ):
for test_tags in ( tags, sibling_tags ):
ok_tags = self._tag_blacklist.Filter( test_tags )
ok_tags = self._tag_blacklist.Filter( test_tags, apply_unnamespaced_rules_to_namespaced_tags = True )
if len( ok_tags ) < len( test_tags ):

91
hydrus/ClientImportSubscriptions.py
View File

@ -184,7 +184,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
return HydrusData.TimeHasPassed( self._no_work_until )
def _QueryFileLoginIsOK( self, query ):
def _QueryFileLoginOK( self, query ):
file_seed_cache = query.GetFileSeedCache()
@ -244,7 +244,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
return result
def _QuerySyncLoginIsOK( self, query ):
def _QuerySyncLoginOK( self, query ):
gallery_seed_log = query.GetGallerySeedLog()
@ -504,11 +504,21 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
p1 = not self._CanDoWorkNow()
p4 = not query.BandwidthIsOK( self._name )
p5 = not self._QueryFileLoginIsOK( query )
p3 = not query.DomainOK()
p4 = not query.BandwidthOK( self._name )
p5 = not self._QueryFileLoginOK( query )
if p1 or p4 or p5:
if p3 and this_query_has_done_work:
job_key.SetVariable( 'popup_text_2', 'domain had errors, will try again later' )
self._DelayWork( 3600, 'domain errors, will try again later' )
time.sleep( 5 )
if p4 and this_query_has_done_work:
job_key.SetVariable( 'popup_text_2', 'no more bandwidth to download files, will do some more later' )
@ -666,16 +676,24 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
if query.HasFileWorkToDo():
if query.BandwidthIsOK( self._name ):
bandwidth_ok = query.BandwidthOK( self._name )
domain_ok = query.DomainOK( self._name )
if HG.subscription_report_mode:
if HG.subscription_report_mode:
HydrusData.ShowText( 'Subscription "{}" checking if any file work due: True'.format( self._name ) )
HydrusData.ShowText( 'Subscription "{}" checking if any file work due: True, bandwidth ok: {}, domain ok: {}'.format( self._name, bandwidth_ok, domain_ok ) )
if bandwidth_ok and domain_ok:
return True
if not domain_ok:
self._DelayWork( 3600, 'domain errors, will try again later' )
if HG.subscription_report_mode:
@ -779,7 +797,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
while gallery_seed_log.WorkToDo():
p1 = not self._CanDoWorkNow()
p3 = not self._QuerySyncLoginIsOK( query )
p3 = not self._QuerySyncLoginOK( query )
if p1 or p3:
@ -1005,7 +1023,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
if this_is_initial_sync:
if not query.BandwidthIsOK( self._name ) and not have_made_an_initial_sync_bandwidth_notification:
if not query.BandwidthOK( self._name ) and not have_made_an_initial_sync_bandwidth_notification:
HydrusData.ShowText( 'FYI: The query "' + query_name + '" for subscription "' + self._name + '" performed its initial sync ok, but that domain is short on bandwidth right now, so no files will be downloaded yet. The subscription will catch up in future as bandwidth becomes available. You can review the estimated time until bandwidth is available under the manage subscriptions dialog. If more queries are performing initial syncs in this run, they may be the same.' )
@ -1846,8 +1864,15 @@ class SubscriptionQuery( HydrusSerialisable.SerialisableBase ):
url = file_seed.file_seed_data
example_nj = ClientNetworkingJobs.NetworkJobSubscription( subscription_key, 'GET', url )
example_network_contexts = example_nj.GetNetworkContexts()
try: # if the url is borked for some reason
example_nj = ClientNetworkingJobs.NetworkJobSubscription( subscription_key, 'GET', url )
example_network_contexts = example_nj.GetNetworkContexts()
except:
return [ ClientNetworkingContexts.NetworkContext( CC.NETWORK_CONTEXT_SUBSCRIPTION, subscription_key ), ClientNetworkingContexts.GLOBAL_NETWORK_CONTEXT ]
return example_network_contexts
@ -1900,20 +1925,20 @@ class SubscriptionQuery( HydrusSerialisable.SerialisableBase ):
def BandwidthIsOK( self, subscription_name ):
def BandwidthOK( self, subscription_name ):
example_network_contexts = self._GetExampleNetworkContexts( subscription_name )
threshold = 90
result = HG.client_controller.network_engine.bandwidth_manager.CanDoWork( example_network_contexts, threshold = threshold )
bandwidth_ok = HG.client_controller.network_engine.bandwidth_manager.CanDoWork( example_network_contexts, threshold = threshold )
if HG.subscription_report_mode:
HydrusData.ShowText( 'Query "' + self.GetHumanName() + '" bandwidth test. Bandwidth ok: ' + str( result ) + '.' )
HydrusData.ShowText( 'Query "' + self.GetHumanName() + '" bandwidth/domain test. Bandwidth ok: {}'.format( bandwidth_ok ) )
return result
return bandwidth_ok
def CanCheckNow( self ):
@ -1940,6 +1965,27 @@ class SubscriptionQuery( HydrusSerialisable.SerialisableBase ):
self._status = ClientImporting.CHECKER_STATUS_OK
def DomainOK( self ):
file_seed = self._file_seed_cache.GetNextFileSeed( CC.STATUS_UNKNOWN )
if file_seed is None:
return True
url = file_seed.file_seed_data
domain_ok = HG.client_controller.network_engine.domain_manager.DomainOK( url )
if HG.subscription_report_mode:
HydrusData.ShowText( 'Query "' + self.GetHumanName() + '" domain test. Domain ok: {}'.format( domain_ok ) )
return domain_ok
def GetBandwidthWaitingEstimate( self, subscription_name ):
example_network_contexts = self._GetExampleNetworkContexts( subscription_name )
@ -2027,7 +2073,16 @@ class SubscriptionQuery( HydrusSerialisable.SerialisableBase ):
if self.HasFileWorkToDo():
file_bandwidth_estimate = self.GetBandwidthWaitingEstimate( subscription_name )
try:
file_bandwidth_estimate = self.GetBandwidthWaitingEstimate( subscription_name )
except:
# this is tricky, but if there is a borked url in here causing trouble, we should let it run and error out immediately tbh
file_bandwidth_estimate = 0
if file_bandwidth_estimate == 0:

62
hydrus/ClientLocalServerResources.py
View File

@ -4,6 +4,7 @@ from . import ClientConstants as CC
from . import ClientImportFileSeeds
from . import ClientMedia
from . import ClientNetworkingContexts
from . import ClientNetworkingDomain
from . import ClientSearch
from . import ClientTags
from . import HydrusConstants as HC
@ -14,7 +15,6 @@ from . import HydrusNetworking
from . import HydrusPaths
from . import HydrusServerResources
from . import HydrusTags
import http.cookiejar
import json
import os
import time
@ -917,7 +917,12 @@ class HydrusResourceClientAPIRestrictedAddTagsAddTags( HydrusResourceClientAPIRe
content_action = int( content_action )
tags = HydrusTags.CleanTags( tags )
tags = list( tags )
if isinstance( tags[0], str ):
tags = HydrusTags.CleanTags( tags )
if len( tags ) == 0:
@ -952,6 +957,8 @@ class HydrusResourceClientAPIRestrictedAddTagsAddTags( HydrusResourceClientAPIRe
if content_action == HC.CONTENT_UPDATE_PETITION:
tags = list( tags )
if isinstance( tags[0], str ):
tags_and_reasons = [ ( tag, 'Petitioned from API' ) for tag in tags ]
@ -1462,6 +1469,7 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
metadata_row[ 'hash' ] = file_info_manager.hash.hex()
metadata_row[ 'size' ] = file_info_manager.size
metadata_row[ 'mime' ] = HC.mime_mimetype_string_lookup[ file_info_manager.mime ]
metadata_row[ 'ext' ] = HC.mime_ext_lookup[ file_info_manager.mime ]
metadata_row[ 'width' ] = file_info_manager.width
metadata_row[ 'height' ] = file_info_manager.height
metadata_row[ 'duration' ] = file_info_manager.duration
@ -1607,6 +1615,9 @@ class HydrusResourceClientAPIRestrictedManageCookiesSetCookies( HydrusResourceCl
cookie_rows = request.parsed_request_args.GetValue( 'cookies', list )
domains_cleared = set()
domains_set = set()
for cookie_row in cookie_rows:
if len( cookie_row ) != 5:
@ -1631,26 +1642,47 @@ class HydrusResourceClientAPIRestrictedManageCookiesSetCookies( HydrusResourceCl
if value is None:
domains_cleared.add( domain )
session.cookies.clear( domain, path, name )
else:
version = 0
port = None
port_specified = False
domain_specified = True
domain_initial_dot = domain.startswith( '.' )
path_specified = True
secure = False
discard = False
comment = None
comment_url = None
rest = {}
domains_set.add( domain )
cookie = http.cookiejar.Cookie( version, name, value, port, port_specified, domain, domain_specified, domain_initial_dot, path, path_specified, secure, expires, discard, comment, comment_url, rest )
ClientNetworkingDomain.AddCookieToSession( session, name, value, domain, path, expires )
session.cookies.set_cookie( cookie )
if HG.client_controller.new_options.GetBoolean( 'notify_client_api_cookies' ) and len( domains_cleared ) + len( domains_set ) > 0:
domains_cleared = list( domains_cleared )
domains_set = list( domains_set )
domains_cleared.sort()
domains_set.sort()
message = 'Cookies sent from API:'
if len( domains_cleared ) > 0:
message = '{} ({} cleared)'.format( message, ', '.join( domains_cleared ) )
if len( domains_set ) > 0:
message = '{} ({} set)'.format( message, ', '.join( domains_set ) )
from . import ClientThreading
job_key = ClientThreading.JobKey()
job_key.SetVariable( 'popup_text_1', message )
job_key.Delete( 5 )
HG.client_controller.pub( 'message', job_key )
HG.client_controller.network_engine.session_manager.SetDirty()

4
hydrus/ClientNetworking.py
View File

@ -384,6 +384,10 @@ class NetworkEngine( object ):
return True
elif not job.DomainOK():
return True
else:
if HG.network_report_mode:

84
hydrus/ClientNetworkingDomain.py
View File

@ -9,12 +9,31 @@ from . import HydrusData
from . import HydrusExceptions
from . import HydrusNetworking
from . import HydrusSerialisable
import http.cookiejar
import os
import re
import threading
import time
import urllib.parse
def AddCookieToSession( session, name, value, domain, path, expires ):
version = 0
port = None
port_specified = False
domain_specified = True
domain_initial_dot = domain.startswith( '.' )
path_specified = True
secure = False
discard = False
comment = None
comment_url = None
rest = {}
cookie = http.cookiejar.Cookie( version, name, value, port, port_specified, domain, domain_specified, domain_initial_dot, path, path_specified, secure, expires, discard, comment, comment_url, rest )
session.cookies.set_cookie( cookie )
def AlphabetiseQueryText( query_text ):
( query_dict, param_order ) = ConvertQueryTextToDict( query_text )
@ -392,6 +411,8 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
self._second_level_domains_to_url_classes = collections.defaultdict( list )
self._second_level_domains_to_network_infrastructure_errors = collections.defaultdict( list )
from . import ClientImportOptions
self._file_post_default_tag_import_options = ClientImportOptions.TagImportOptions()
@ -1173,6 +1194,52 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
self.SetURLClasses( url_classes )
def DomainOK( self, url ):
with self._lock:
try:
domain = ConvertURLIntoSecondLevelDomain( url )
except:
return True
number_of_errors = HG.client_controller.new_options.GetInteger( 'domain_network_infrastructure_error_number' )
error_time_delta = HG.client_controller.new_options.GetInteger( 'domain_network_infrastructure_error_time_delta' )
if number_of_errors == 0:
return True
# this will become flexible and customisable when I have domain profiles/status/ui
# also should extend it to 'global', so if multiple domains are having trouble, we maybe assume the whole connection is down? it would really be nicer to have a better sockets-level check there
if domain in self._second_level_domains_to_network_infrastructure_errors:
network_infrastructure_errors = self._second_level_domains_to_network_infrastructure_errors[ domain ]
network_infrastructure_errors = [ timestamp for timestamp in network_infrastructure_errors if not HydrusData.TimeHasPassed( timestamp + error_time_delta ) ]
self._second_level_domains_to_network_infrastructure_errors[ domain ] = network_infrastructure_errors
if len( network_infrastructure_errors ) >= number_of_errors:
return False
elif len( network_infrastructure_errors ) == 0:
del self._second_level_domains_to_network_infrastructure_errors[ domain ]
return True
def GenerateValidationPopupProcess( self, network_contexts ):
with self._lock:
@ -1630,6 +1697,23 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
def ReportNetworkInfrastructureError( self, url ):
with self._lock:
try:
domain = ConvertURLIntoDomain( url )
except:
return
self._second_level_domains_to_network_infrastructure_errors[ domain ].append( HydrusData.GetNow() )
def SetClean( self ):
with self._lock:

140
hydrus/ClientNetworkingJobs.py
View File

@ -10,13 +10,22 @@ from . import HydrusNetworking
from . import HydrusThreading
from . import HydrusText
import os
import re
import requests
import threading
import traceback
import time
import urllib
try:
import cloudscraper
CLOUDSCRAPER_OK = True
except:
CLOUDSCRAPER_OK = False
def ConvertStatusCodeAndDataIntoExceptionInfo( status_code, data, is_hydrus_service = False ):
( error_text, encoding ) = HydrusText.NonFailingUnicodeDecode( data, 'utf-8' )
@ -128,6 +137,7 @@ class NetworkJob( object ):
self._for_login = False
self._current_connection_attempt_number = 1
self._we_tried_cloudflare_once = False
self._additional_headers = {}
@ -506,6 +516,75 @@ class NetworkJob( object ):
self._wake_time = HydrusData.GetNow() + seconds
def _SolveCloudFlare( self, response ):
if CLOUDSCRAPER_OK:
try:
is_firewall = cloudscraper.CloudScraper.is_Firewall_Blocked( response )
is_attemptable = cloudscraper.CloudScraper.is_reCaptcha_Challenge( response ) or cloudscraper.CloudScraper.is_IUAM_Challenge( response )
except Exception as e:
HydrusData.Print( 'cloudflarescraper had an error looking at "{}" response: {}'.format( self._url, str( e ) ) )
HydrusData.PrintException( e )
return
if is_firewall:
raise HydrusExceptions.CloudFlareException( 'It looks like the site has Firewall-Blocked your IP or IP range with CloudFlare.' )
if is_attemptable:
try:
with self._lock:
ncs = list( self._network_contexts )
snc = self._session_network_context
headers = self.engine.domain_manager.GetHeaders( ncs )
if 'User-Agent' not in headers:
raise HydrusExceptions.CloudFlareException( 'No User-Agent set for hydrus!' )
user_agent = headers[ 'User-Agent' ]
( cf_tokens, user_agent ) = cloudscraper.get_tokens( self._url, browser = { 'custom' : user_agent } )
session = self.engine.session_manager.GetSession( snc )
domain = '.{}'.format( ClientNetworkingDomain.ConvertURLIntoSecondLevelDomain( self._url ) )
path = '/'
expires = 30 * 86400
for ( name, value ) in cf_tokens.items():
ClientNetworkingDomain.AddCookieToSession( session, name, value, domain, path, expires )
self.engine.session_manager.SetDirty()
except Exception as e:
HydrusData.PrintException( e )
raise HydrusExceptions.CloudFlareException( 'Looks like an unsolvable CloudFlare issue: {}'.format( str( e ) ) )
raise HydrusExceptions.ShouldReattemptNetworkException( 'CloudFlare needed solving.' )
def _WaitOnConnectionError( self, status_text ):
connection_error_wait_time = HG.client_controller.new_options.GetInteger( 'connection_error_wait_time' )
@ -686,6 +765,28 @@ class NetworkJob( object ):
def DomainOK( self ):
with self._lock:
if self._max_connection_attempts_allowed == 1:
return True
domain_ok = self.engine.domain_manager.DomainOK( self._url )
if not domain_ok:
self._status_text = 'This domain has had several serious errors recently. Waiting a bit.'
self._Sleep( 10 )
return domain_ok
def GenerateLoginProcess( self ):
with self._lock:
@ -1089,6 +1190,14 @@ class NetworkJob( object ):
self._status_text = str( response.status_code ) + ' - ' + str( response.reason )
# it is important we do this before ReadResponse, as the CF test needs r.text, which is nullified if we first access with iter_content
if not self._we_tried_cloudflare_once:
self._we_tried_cloudflare_once = True
self._SolveCloudFlare( response )
self._ReadResponse( response, self._stream_io, 104857600 )
with self._lock:
@ -1114,9 +1223,13 @@ class NetworkJob( object ):
self._current_connection_attempt_number += 1
if not self._CanReattemptRequest():
if self._CanReattemptRequest():
raise HydrusExceptions.NetworkException( 'Server reported very limited bandwidth: ' + str( e ) )
self.engine.domain_manager.ReportNetworkInfrastructureError( self._url )
else:
raise HydrusExceptions.BandwidthException( 'Server reported very limited bandwidth: ' + str( e ) )
self._WaitOnServersideBandwidth( 'server reported limited bandwidth' )
@ -1127,10 +1240,10 @@ class NetworkJob( object ):
if not self._CanReattemptRequest():
raise HydrusExceptions.NetworkException( 'Ran out of reattempts on this error: ' + str( e ) )
raise HydrusExceptions.NetworkInfrastructureException( 'Ran out of reattempts on this error: ' + str( e ) )
self._WaitOnConnectionError( 'server suddenly stopped delivering data' )
self._WaitOnConnectionError( str( e ) )
except requests.exceptions.ChunkedEncodingError:
@ -1138,7 +1251,7 @@ class NetworkJob( object ):
if not self._CanReattemptRequest():
raise HydrusExceptions.ConnectionException( 'Unable to complete request--it broke mid-way!' )
raise HydrusExceptions.StreamTimeoutException( 'Unable to complete request--it broke mid-way!' )
self._WaitOnConnectionError( 'connection broke mid-request' )
@ -1147,7 +1260,11 @@ class NetworkJob( object ):
self._current_connection_attempt_number += 1
if not self._CanReattemptConnection():
if self._CanReattemptConnection():
self.engine.domain_manager.ReportNetworkInfrastructureError( self._url )
else:
raise HydrusExceptions.ConnectionException( 'Could not connect!' )
@ -1160,7 +1277,7 @@ class NetworkJob( object ):
if not self._CanReattemptRequest():
raise HydrusExceptions.ConnectionException( 'Connection successful, but reading response timed out!' )
raise HydrusExceptions.StreamTimeoutException( 'Connection successful, but reading response timed out!' )
self._WaitOnConnectionError( 'read timed out' )
@ -1182,11 +1299,16 @@ class NetworkJob( object ):
trace = traceback.format_exc()
if not isinstance( e, ( HydrusExceptions.ConnectionException, HydrusExceptions.SizeException ) ):
if not isinstance( e, ( HydrusExceptions.NetworkInfrastructureException, HydrusExceptions.StreamTimeoutException, HydrusExceptions.SizeException ) ):
HydrusData.Print( trace )
if isinstance( e, HydrusExceptions.NetworkInfrastructureException ):
self.engine.domain_manager.ReportNetworkInfrastructureError( self._url )
self._status_text = 'Error: ' + str( e )
self._SetError( e, trace )

1
hydrus/ClientNetworkingSessions.py
View File

@ -19,7 +19,6 @@ except:
SOCKS_PROXY_OK = False
class NetworkSessionManager( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER

7
hydrus/ClientOptions.py
View File

@ -223,6 +223,10 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
self._dictionary[ 'booleans' ][ 'close_client_to_system_tray' ] = False
self._dictionary[ 'booleans' ][ 'start_client_in_system_tray' ] = False
self._dictionary[ 'booleans' ][ 'use_qt_file_dialogs' ] = False
self._dictionary[ 'booleans' ][ 'notify_client_api_cookies' ] = False
#
self._dictionary[ 'colours' ] = HydrusSerialisable.SerialisableDictionary()
@ -351,6 +355,9 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
self._dictionary[ 'integers' ][ 'animated_scanbar_height' ] = 20
self._dictionary[ 'integers' ][ 'animated_scanbar_nub_width' ] = 10
self._dictionary[ 'integers' ][ 'domain_network_infrastructure_error_number' ] = 3
self._dictionary[ 'integers' ][ 'domain_network_infrastructure_error_time_delta' ] = 600
#
self._dictionary[ 'keys' ] = {}

19
hydrus/ClientTags.py
View File

@ -413,7 +413,7 @@ class TagFilter( HydrusSerialisable.SerialisableBase ):
return NotImplemented
def _GetTagSlices( self, tag ):
def _GetTagSlices( self, tag, apply_unnamespaced_rules_to_namespaced_tags ):
( namespace, subtag ) = HydrusTags.SplitTag( tag )
@ -421,6 +421,11 @@ class TagFilter( HydrusSerialisable.SerialisableBase ):
tag_slices.append( tag )
if tag != subtag and apply_unnamespaced_rules_to_namespaced_tags:
tag_slices.append( subtag )
if namespace != '':
tag_slices.append( namespace + ':' )
@ -444,9 +449,9 @@ class TagFilter( HydrusSerialisable.SerialisableBase ):
self._tag_slices_to_rules = dict( serialisable_info )
def _TagOK( self, tag ):
def _TagOK( self, tag, apply_unnamespaced_rules_to_namespaced_tags = False ):
tag_slices = self._GetTagSlices( tag )
tag_slices = self._GetTagSlices( tag, apply_unnamespaced_rules_to_namespaced_tags = apply_unnamespaced_rules_to_namespaced_tags )
blacklist_encountered = False
@ -493,11 +498,11 @@ class TagFilter( HydrusSerialisable.SerialisableBase ):
def Filter( self, tags ):
def Filter( self, tags, apply_unnamespaced_rules_to_namespaced_tags = False ):
with self._lock:
return { tag for tag in tags if self._TagOK( tag ) }
return { tag for tag in tags if self._TagOK( tag, apply_unnamespaced_rules_to_namespaced_tags = apply_unnamespaced_rules_to_namespaced_tags ) }
@ -517,11 +522,11 @@ class TagFilter( HydrusSerialisable.SerialisableBase ):
def TagOK( self, tag ):
def TagOK( self, tag, apply_unnamespaced_rules_to_namespaced_tags = False ):
with self._lock:
return self._TagOK( tag )
return self._TagOK( tag, apply_unnamespaced_rules_to_namespaced_tags = apply_unnamespaced_rules_to_namespaced_tags )

2
hydrus/HydrusConstants.py
View File

@ -70,7 +70,7 @@ options = {}
# Misc
NETWORK_VERSION = 18
SOFTWARE_VERSION = 392
SOFTWARE_VERSION = 393
CLIENT_API_VERSION = 11
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )

5
hydrus/HydrusExceptions.py
View File

@ -52,8 +52,11 @@ class NetworkInfrastructureException( NetworkException ): pass
class ConnectionException( NetworkInfrastructureException ): pass
class FirewallException( NetworkInfrastructureException ): pass
class ServerBusyException( NetworkInfrastructureException ): pass
class CloudFlareException( NetworkInfrastructureException ): pass
class BandwidthException( NetworkInfrastructureException ): pass
class StreamTimeoutException( NetworkException ): pass
class BandwidthException( NetworkException ): pass
class NetworkVersionException( NetworkException ): pass
class NoContentException( NetworkException ): pass
class NotFoundException( NetworkException ): pass

52
hydrus/QtPorting.py
View File

@ -213,7 +213,16 @@ class DirPickerCtrl( QW.QWidget ):
existing_path = self._path_edit.text()
path = QW.QFileDialog.getExistingDirectory( self, '', existing_path )
if HG.client_controller.new_options.GetBoolean( 'use_qt_file_dialogs' ):
options = QW.QFileDialog.Options( QW.QFileDialog.DontUseNativeDialog )
else:
options = QW.QFileDialog.Options()
path = QW.QFileDialog.getExistingDirectory( self, '', existing_path, options = options )
if path == '':
@ -292,26 +301,35 @@ class FilePickerCtrl( QW.QWidget ):
existing_path = self._starting_directory
if HG.client_controller.new_options.GetBoolean( 'use_qt_file_dialogs' ):
options = QW.QFileDialog.Options( QW.QFileDialog.DontUseNativeDialog )
else:
options = QW.QFileDialog.Options()
if self._save_mode:
if self._wildcard:
path = QW.QFileDialog.getSaveFileName( self, '', existing_path, filter = self._wildcard, selectedFilter = self._wildcard )[0]
path = QW.QFileDialog.getSaveFileName( self, '', existing_path, filter = self._wildcard, selectedFilter = self._wildcard, options = options )[0]
else:
path = QW.QFileDialog.getSaveFileName( self, '', existing_path )[0]
path = QW.QFileDialog.getSaveFileName( self, '', existing_path, options = options )[0]
else:
if self._wildcard:
path = QW.QFileDialog.getOpenFileName( self, '', existing_path, filter = self._wildcard, selectedFilter = self._wildcard )[0]
path = QW.QFileDialog.getOpenFileName( self, '', existing_path, filter = self._wildcard, selectedFilter = self._wildcard, options = options )[0]
else:
path = QW.QFileDialog.getOpenFileName( self, '', existing_path )[0]
path = QW.QFileDialog.getOpenFileName( self, '', existing_path, options = options )[0]
@ -1637,7 +1655,7 @@ class CheckListBox( QW.QListWidget ):
self.itemClicked.connect( self._ItemCheckStateChanged )
self.setSelectionMode( QW.QAbstractItemView.ExtendedSelection )
def Check(self, index, state = True):
@ -2058,6 +2076,11 @@ class DirDialog( QW.QFileDialog ):
self.setOption( QW.QFileDialog.ShowDirsOnly, True )
if HG.client_controller.new_options.GetBoolean( 'use_qt_file_dialogs' ):
self.setOption( QW.QFileDialog.DontUseNativeDialog, True )
def __enter__( self ):
@ -2087,7 +2110,7 @@ class DirDialog( QW.QFileDialog ):
class FileDialog( QW.QFileDialog ):
def __init__( self, parent = None, message = None, acceptMode = QW.QFileDialog.AcceptOpen, fileMode = QW.QFileDialog.ExistingFile, defaultFile = None, wildcard = None ):
QW.QFileDialog.__init__( self, parent )
@ -2101,7 +2124,12 @@ class FileDialog( QW.QFileDialog ):
if defaultFile: self.setDirectory( defaultFile )
if wildcard: self.setNameFilter( wildcard )
if HG.client_controller.new_options.GetBoolean( 'use_qt_file_dialogs' ):
self.setOption( QW.QFileDialog.DontUseNativeDialog, True )
def __enter__( self ):

1
hydrus/TestClientAPI.py
View File

@ -1624,6 +1624,7 @@ class TestClientAPI( unittest.TestCase ):
metadata_row[ 'hash' ] = file_info_manager.hash.hex()
metadata_row[ 'size' ] = file_info_manager.size
metadata_row[ 'mime' ] = HC.mime_mimetype_string_lookup[ file_info_manager.mime ]
metadata_row[ 'ext' ] = HC.mime_ext_lookup[ file_info_manager.mime ]
metadata_row[ 'width' ] = file_info_manager.width
metadata_row[ 'height' ] = file_info_manager.height
metadata_row[ 'duration' ] = file_info_manager.duration

35
hydrus/TestHydrusSerialisable.py
View File

@ -661,4 +661,39 @@ class TestSerialisables( unittest.TestCase ):
self.assertEqual( tag_filter.Filter( tags ), { 'title:test title', 'series:neon genesis evangelion', 'series:kill la kill', 'blue eyes' } )
# blacklist namespace test
blacklist_tags = { 'nintendo', 'studio:nintendo' }
#
tag_filter = ClientTags.TagFilter()
tag_filter.SetRule( 'nintendo', CC.FILTER_BLACKLIST )
self._dump_and_load_and_test( tag_filter, test )
self.assertEqual( tag_filter.Filter( blacklist_tags ), { 'studio:nintendo' } )
#
tag_filter = ClientTags.TagFilter()
tag_filter.SetRule( 'nintendo', CC.FILTER_BLACKLIST )
self._dump_and_load_and_test( tag_filter, test )
self.assertEqual( tag_filter.Filter( blacklist_tags, apply_unnamespaced_rules_to_namespaced_tags = True ), set() )
#
tag_filter = ClientTags.TagFilter()
tag_filter.SetRule( 'nintendo', CC.FILTER_BLACKLIST )
tag_filter.SetRule( 'studio:nintendo', CC.FILTER_WHITELIST )
self._dump_and_load_and_test( tag_filter, test )
self.assertEqual( tag_filter.Filter( blacklist_tags, apply_unnamespaced_rules_to_namespaced_tags = True ), { 'studio:nintendo' } )

1
requirements.txt
View File

@ -1,5 +1,6 @@
beautifulsoup4==4.8.2
chardet==3.0.4
cloudscraper==1.2.33
html5lib==1.0.1
httplib2==0.9.2
lxml==4.5.0

BIN
static/default/parsers/deviant art file extended_fetch parser.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.6 KiB

BIN
static/default/parsers/deviant art flash sandbox page parser.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.2 KiB

BIN
static/default/url_classes/deviant art file page extended_fetch api.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.2 KiB

BIN
static/default/url_classes/deviant art file page.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.7 KiB

After

Width:  |  Height:  |  Size: 1.8 KiB

BIN
static/default/url_classes/deviant art flash sandbox page.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.0 KiB

4
static/mpv-conf/default_mpv.conf
View File

@ -5,6 +5,10 @@ autoload-files=no
access-references=no
rescan-external-files=keep-selection
# Some OSes immediately hide the mouse cursor
cursor-autohide=no
# seems to work well for dynamic audio normalisation
af=lavfi=[loudnorm=I=-16:TP=-3:LRA=4]