RepoMirrors
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Version 456 

closes #976, closes #933, closes #922, closes #910
This commit is contained in:
Hydrus Network Developer 2021-09-29 16:20:29 -05:00
parent 6151866d29
commit 6cd2995275
29 changed files with 2262 additions and 1687 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
help/changelog.html
View File

@ -8,6 +8,27 @@
<div class="content">
<h3 id="changelog"><a href="#changelog">changelog</a></h3>
<ul>
<li><h3 id="version_456"><a href="#version_456">version 456</a></h3></li>
<ul>
<li>misc:</li>
<li>the client no longer regularly commits a full garbage collection during memory maintenance. this debug-tier operation can take up to 15s on very large clients, resulting in awful lag. various instances of forcing it after big operations complete (e.g. to encourage post-subscription memory cleanup), are now replaced with regular pauses to allow python to clean itself more granularly. this may result in temporary memory bloat for some very subscription-heavy clients, so feedback would be appreciated</li>
<li>right-clicking on a single url import item in a 'file log' now shows you all the known hashes, parsed urls, and parsed tags for that item. I hope this will help debug some weird problems!</li>
<li>all multi-column lists across the program now convert an enter/return key press into an 'activate' command, as if you had double-clicked. this should make it easier to, for instance, highlight a downloader or shift/ctrl select a bunch of sibling rows and mass-delete (issue #933)</li>
<li>the subscription gap filler button now propagates file import options and tag import options from the subscription to the downloader it creates (issue #910)</li>
<li>a new 'mpv report mode' now prints a huge amount of mpv debug information to the hydrus log when activated</li>
<li>improved how mpv prints log messages to the hydrus log, including immediate log flushing</li>
<li>fixed a bug that meant the hydrus server was not saving custom update period or anonymisation period for next boot. thank you for the reports, and sorry for the trouble! (issue #976)</li>
<li>cleaned up some database savepoint handling after a serious transaction error occurs</li>
<li>the client api now ignores any parameter with a value of null, as if it were not there, rather than moaning about invalid datatypes (issue #922)</li>
<li>.</li>
<li>url classes:</li>
<li>the edit url class dialog is now broken into two notebook pages--'match rules', which strictly covers how to recognise a url, and 'options', which handles url storage, conversion, and normalisation</li>
<li>url classes can now support single-value parameters (a parameter with just a value, not a key/value pair). if turned on, then at least one single-value parameter is required to match the url, and multiple are permitted. a checkbox in the dialog turns this on and a string match lets you determine if the url class matches the received single value params</li>
<li>added unit tests to test the new single-value parameter matching</li>
<li>fixed an issue where StringMatch buttons were not emitting their valueChanged signal, guess how I discovered that bug this week</li>
<li>fixed the insertion of default parameter values when the URL Class has non-alphabetised params</li>
<li>refactored and cleaned up some related parsing and string convertion code into new ClientString module</li>
</ul>
<li><h3 id="version_455"><a href="#version_455">version 455</a></h3></li>
<ul>
<li>misc:</li>

3
hydrus/client/ClientController.py
View File

@ -1,4 +1,3 @@
import gc
import hashlib
import os
import psutil
@ -2030,8 +2029,6 @@ class Controller( HydrusController.HydrusController ):
try:
gc.collect()
self.frame_splash_status.SetTitleText( 'shutting down gui\u2026' )
self.ShutdownView()

16
hydrus/client/ClientFiles.py
View File

@ -1,5 +1,4 @@
import collections
import gc
import os
import random
import threading
@ -1321,8 +1320,6 @@ class FilesMaintenanceManager( object ):
self._idle_work_rules = HydrusNetworking.BandwidthRules()
self._active_work_rules = HydrusNetworking.BandwidthRules()
self._jobs_since_last_gc_collect = 0
self._ReInitialiseWorkRules()
self._maintenance_lock = threading.Lock()
@ -1748,6 +1745,8 @@ class FilesMaintenanceManager( object ):
try:
big_pauser = HydrusData.BigJobPauser( wait_time = 0.8 )
cleared_jobs = []
num_to_do = len( media_results )
@ -1759,6 +1758,8 @@ class FilesMaintenanceManager( object ):
for ( i, media_result ) in enumerate( media_results ):
big_pauser.Pause()
hash = media_result.GetHash()
if job_key.IsCancelled():
@ -1865,15 +1866,6 @@ class FilesMaintenanceManager( object ):
cleared_jobs.append( ( hash, job_type, additional_data ) )
self._jobs_since_last_gc_collect += 1
if self._jobs_since_last_gc_collect > 100:
gc.collect()
self._jobs_since_last_gc_collect = 0
if len( cleared_jobs ) > 100:
self._controller.WriteSynchronous( 'file_maintenance_clear_jobs', cleared_jobs )

1295
hydrus/client/ClientParsing.py
View File

File diff suppressed because it is too large Load Diff

1246
hydrus/client/ClientStrings.py Normal file
View File

File diff suppressed because it is too large Load Diff

39
hydrus/client/gui/ClientGUI.py
View File

@ -1361,7 +1361,13 @@ class FrameGUI( ClientGUITopLevelWindows.MainFrameThatResizes ):
job_key.SetStatusTitle( 'sub gap downloader test' )
call = HydrusData.Call( HG.client_controller.pub, 'make_new_subscription_gap_downloader', ( b'', 'safebooru tag search' ), 'skirt', 2 )
file_import_options = HG.client_controller.new_options.GetDefaultFileImportOptions( 'quiet' )
from hydrus.client.importing.options import TagImportOptions
tag_import_options = TagImportOptions.TagImportOptions( is_default = True )
call = HydrusData.Call( HG.client_controller.pub, 'make_new_subscription_gap_downloader', ( b'', 'safebooru tag search' ), 'skirt', file_import_options, tag_import_options, 2 )
call.SetLabel( 'start a new downloader for this to fill in the gap!' )
@ -4842,22 +4848,22 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
def _SwitchBoolean( self, name ):
if name == 'callto_report_mode':
HG.callto_report_mode = not HG.callto_report_mode
elif name == 'daemon_report_mode':
HG.daemon_report_mode = not HG.daemon_report_mode
elif name == 'cache_report_mode':
if name == 'cache_report_mode':
HG.cache_report_mode = not HG.cache_report_mode
elif name == 'callto_report_mode':
HG.callto_report_mode = not HG.callto_report_mode
elif name == 'canvas_tile_outline_mode':
HG.canvas_tile_outline_mode = not HG.canvas_tile_outline_mode
elif name == 'daemon_report_mode':
HG.daemon_report_mode = not HG.daemon_report_mode
elif name == 'db_report_mode':
HG.db_report_mode = not HG.db_report_mode
@ -4886,6 +4892,14 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
HG.media_load_report_mode = not HG.media_load_report_mode
elif name == 'mpv_report_mode':
HG.mpv_report_mode = not HG.mpv_report_mode
level = 'debug' if HG.mpv_report_mode else 'fatal'
self._controller.pub( 'set_mpv_log_level', level )
elif name == 'network_report_mode':
HG.network_report_mode = not HG.network_report_mode
@ -5292,7 +5306,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
event.ignore() # we always ignore, as we'll close through the window through other means
def CreateNewSubscriptionGapDownloader( self, gug_key_and_name, query_text, file_limit ):
def CreateNewSubscriptionGapDownloader( self, gug_key_and_name, query_text, file_import_options, tag_import_options, file_limit ):
page = self._notebook.GetOrMakeGalleryDownloaderPage( desired_page_name = 'subscription gap downloaders', select_page = True )
@ -5305,7 +5319,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
multiple_gallery_import = management_controller.GetVariable( 'multiple_gallery_import' )
multiple_gallery_import.PendSubscriptionGapDownloader( gug_key_and_name, query_text, file_limit )
multiple_gallery_import.PendSubscriptionGapDownloader( gug_key_and_name, query_text, file_import_options, tag_import_options, file_limit )
self._notebook.ShowPage( page )
@ -6094,6 +6108,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
ClientGUIMenus.AppendMenuCheckItem( report_modes, 'gui report mode', 'Have the gui report inside information, where supported.', HG.gui_report_mode, self._SwitchBoolean, 'gui_report_mode' )
ClientGUIMenus.AppendMenuCheckItem( report_modes, 'hover window report mode', 'Have the hover windows report their show/hide logic.', HG.hover_window_report_mode, self._SwitchBoolean, 'hover_window_report_mode' )
ClientGUIMenus.AppendMenuCheckItem( report_modes, 'media load report mode', 'Have the client report media load information, where supported.', HG.media_load_report_mode, self._SwitchBoolean, 'media_load_report_mode' )
ClientGUIMenus.AppendMenuCheckItem( report_modes, 'mpv report mode', 'Have the client report significant mpv debug information.', HG.mpv_report_mode, self._SwitchBoolean, 'mpv_report_mode' )
ClientGUIMenus.AppendMenuCheckItem( report_modes, 'network report mode', 'Have the network engine report new jobs.', HG.network_report_mode, self._SwitchBoolean, 'network_report_mode' )
ClientGUIMenus.AppendMenuCheckItem( report_modes, 'pubsub report mode', 'Report info about every pubsub processed.', HG.pubsub_report_mode, self._SwitchBoolean, 'pubsub_report_mode' )
ClientGUIMenus.AppendMenuCheckItem( report_modes, 'similar files metadata generation report mode', 'Have the phash generation routine report its progress.', HG.phash_generation_report_mode, self._SwitchBoolean, 'phash_generation_report_mode' )

329
hydrus/client/gui/ClientGUIDownloaders.py
View File

@ -12,8 +12,8 @@ from hydrus.core import HydrusSerialisable
from hydrus.client import ClientConstants as CC
from hydrus.client import ClientDefaults
from hydrus.client import ClientParsing
from hydrus.client import ClientPaths
from hydrus.client import ClientStrings
from hydrus.client.gui import ClientGUIDialogs
from hydrus.client.gui import ClientGUIDialogsQuick
from hydrus.client.gui import ClientGUIFunctions
@ -891,11 +891,13 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent: QW.QWidget, url_class: ClientNetworkingDomain.URLClass ):
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
self._update_already_in_progress = False # Used to avoid infinite recursion on control updates.
self._original_url_class = url_class
( url_type, preferred_scheme, netloc, path_components, parameters, api_lookup_converter, send_referral_url, referral_url_converter, example_url ) = url_class.ToTuple()
self._name = QW.QLineEdit( self )
self._url_type = ClientGUICommon.BetterChoice( self )
@ -905,22 +907,22 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
self._url_type.addItem( HC.url_type_string_lookup[ url_type ], url_type )
self._preferred_scheme = ClientGUICommon.BetterChoice( self )
self._notebook = ClientGUICommon.BetterNotebook( self )
#
self._matching_panel = ClientGUICommon.StaticBox( self._notebook, 'matching' )
#
self._preferred_scheme = ClientGUICommon.BetterChoice( self._matching_panel )
self._preferred_scheme.addItem( 'http', 'http' )
self._preferred_scheme.addItem( 'https', 'https' )
self._netloc = QW.QLineEdit( self )
self._netloc = QW.QLineEdit( self._matching_panel )
self._alphabetise_get_parameters = QW.QCheckBox( self )
tt = 'Normally, to ensure the same URLs are merged, hydrus will alphabetise GET parameters as part of the normalisation process.'
tt += os.linesep * 2
tt += 'Almost all servers support GET params in any order. One or two do not. Uncheck this if you know there is a problem.'
self._alphabetise_get_parameters.setToolTip( tt )
self._match_subdomains = QW.QCheckBox( self )
self._match_subdomains = QW.QCheckBox( self._matching_panel )
tt = 'Should this class apply to subdomains as well?'
tt += os.linesep * 2
@ -930,51 +932,15 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
self._match_subdomains.setToolTip( tt )
self._keep_matched_subdomains = QW.QCheckBox( self )
tt = 'Should this url keep its matched subdomains when it is normalised?'
tt += os.linesep * 2
tt += 'This is typically useful for direct file links that are often served on a numbered CDN subdomain like \'img3.example.com\' but are also valid on the neater main domain.'
self._keep_matched_subdomains.setToolTip( tt )
self._can_produce_multiple_files = QW.QCheckBox( self )
tt = 'If checked, the client will not rely on instances of this URL class to predetermine \'already in db\' or \'previously deleted\' outcomes. This is important for post types like pixiv pages (which can ultimately be manga, and represent many pages) and tweets (which can have multiple images).'
tt += os.linesep * 2
tt += 'Most booru-type Post URLs only produce one file per URL and should not have this checked. Checking this avoids some bad logic where the client would falsely think it if it had seen one file at the URL, it had seen them all, but it then means the client has to download those pages\' content again whenever it sees them (so it can check against the direct File URLs, which are always considered one-file each).'
self._can_produce_multiple_files.setToolTip( tt )
self._should_be_associated_with_files = QW.QCheckBox( self )
tt = 'If checked, the client will try to remember this url with any files it ends up importing. It will present this url in \'known urls\' ui across the program.'
tt += os.linesep * 2
tt += 'If this URL is a File or Post URL and the client comes across it after having already downloaded it once, it can skip the redundant download since it knows it already has (or has already deleted) the file once before.'
tt += os.linesep * 2
tt += 'Turning this on is only useful if the URL is non-ephemeral (i.e. the URL will produce the exact same file(s) in six months\' time). It is usually not appropriate for booru gallery or thread urls, which alter regularly, but is for static Post URLs or some fixed doujin galleries.'
self._should_be_associated_with_files.setToolTip( tt )
self._keep_fragment = QW.QCheckBox( self )
tt = 'If checked, fragment text will be kept. This is the component sometimes after an URL that starts with a "#", such as "#kwGFb3xhA3k8B".'
tt += os.linesep * 2
tt += 'This data is never sent to a server, so in normal cases should never be kept, but for some clever services such as Mega, with complicated javascript navigation, it may contain unique clientside navigation data if you open the URL in your browser.'
tt += os.linesep * 2
tt += 'Only turn this on if you know it is needed. For almost all sites, it only hurts the normalisation process.'
self._keep_fragment.setToolTip( tt )
#
path_components_panel = ClientGUICommon.StaticBox( self, 'path components' )
path_components_panel = ClientGUICommon.StaticBox( self._matching_panel, 'path components' )
self._path_components = ClientGUIListBoxes.QueueListBox( path_components_panel, 6, self._ConvertPathComponentRowToString, self._AddPathComponent, self._EditPathComponent )
#
parameters_panel = ClientGUICommon.StaticBox( self, 'parameters' )
parameters_panel = ClientGUICommon.StaticBox( self._matching_panel, 'parameters' )
parameters_listctrl_panel = ClientGUIListCtrl.BetterListCtrlPanel( parameters_panel )
@ -988,7 +954,125 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
#
headers_panel = ClientGUICommon.StaticBox( self, 'header overrides' )
( has_single_value_parameters, single_value_parameters_string_match ) = url_class.GetSingleValueParameterData()
self._has_single_value_parameters = QW.QCheckBox( self._matching_panel )
tt = 'Some URLs have parameters with just a key or a value, not a "key=value" pair. Normally these are removed on normalisation, but if you turn this on, then this URL will keep them and require at least one.'
self._has_single_value_parameters.setToolTip( tt )
self._has_single_value_parameters.setChecked( has_single_value_parameters )
self._single_value_parameters_string_match = ClientGUIStringControls.StringMatchButton( self._matching_panel, single_value_parameters_string_match )
tt = 'All single-value parameters must match this!'
#
self._notebook.addTab( self._matching_panel, 'match rules' )
#
self._options_panel = ClientGUICommon.StaticBox( self._notebook, 'options' )
#
self._keep_matched_subdomains = QW.QCheckBox( self._options_panel )
tt = 'Should this url keep its matched subdomains when it is normalised?'
tt += os.linesep * 2
tt += 'This is typically useful for direct file links that are often served on a numbered CDN subdomain like \'img3.example.com\' but are also valid on the neater main domain.'
self._keep_matched_subdomains.setToolTip( tt )
self._alphabetise_get_parameters = QW.QCheckBox( self._options_panel )
tt = 'Normally, to ensure the same URLs are merged, hydrus will alphabetise GET parameters as part of the normalisation process.'
tt += os.linesep * 2
tt += 'Almost all servers support GET params in any order. One or two do not. Uncheck this if you know there is a problem.'
self._alphabetise_get_parameters.setToolTip( tt )
self._can_produce_multiple_files = QW.QCheckBox( self._options_panel )
tt = 'If checked, the client will not rely on instances of this URL class to predetermine \'already in db\' or \'previously deleted\' outcomes. This is important for post types like pixiv pages (which can ultimately be manga, and represent many pages) and tweets (which can have multiple images).'
tt += os.linesep * 2
tt += 'Most booru-type Post URLs only produce one file per URL and should not have this checked. Checking this avoids some bad logic where the client would falsely think it if it had seen one file at the URL, it had seen them all, but it then means the client has to download those pages\' content again whenever it sees them (so it can check against the direct File URLs, which are always considered one-file each).'
self._can_produce_multiple_files.setToolTip( tt )
self._should_be_associated_with_files = QW.QCheckBox( self._options_panel )
tt = 'If checked, the client will try to remember this url with any files it ends up importing. It will present this url in \'known urls\' ui across the program.'
tt += os.linesep * 2
tt += 'If this URL is a File or Post URL and the client comes across it after having already downloaded it once, it can skip the redundant download since it knows it already has (or has already deleted) the file once before.'
tt += os.linesep * 2
tt += 'Turning this on is only useful if the URL is non-ephemeral (i.e. the URL will produce the exact same file(s) in six months\' time). It is usually not appropriate for booru gallery or thread urls, which alter regularly, but is for static Post URLs or some fixed doujin galleries.'
self._should_be_associated_with_files.setToolTip( tt )
self._keep_fragment = QW.QCheckBox( self._options_panel )
tt = 'If checked, fragment text will be kept. This is the component sometimes after an URL that starts with a "#", such as "#kwGFb3xhA3k8B".'
tt += os.linesep * 2
tt += 'This data is never sent to a server, so in normal cases should never be kept, but for some clever services such as Mega, with complicated javascript navigation, it may contain unique clientside navigation data if you open the URL in your browser.'
tt += os.linesep * 2
tt += 'Only turn this on if you know it is needed. For almost all sites, it only hurts the normalisation process.'
self._keep_fragment.setToolTip( tt )
#
self._referral_url_panel = ClientGUICommon.StaticBox( self._options_panel, 'referral url' )
self._send_referral_url = ClientGUICommon.BetterChoice( self._referral_url_panel )
for send_referral_url_type in ClientNetworkingDomain.SEND_REFERRAL_URL_TYPES:
self._send_referral_url.addItem( ClientNetworkingDomain.send_referral_url_string_lookup[ send_referral_url_type ], send_referral_url_type )
tt = 'Do not change this unless you know you need to. It fixes complicated problems.'
self._send_referral_url.setToolTip( tt )
self._referral_url_converter = ClientGUIStringControls.StringConverterButton( self._referral_url_panel, referral_url_converter )
tt = 'This will generate a referral URL from the original URL. If the URL needs a referral URL, and you can infer what that would be from just this URL, this will let hydrus download this URL without having to previously visit the referral URL (e.g. letting the user drag-and-drop import). It also lets you set up alternate referral URLs for perculiar situations.'
self._referral_url_converter.setToolTip( tt )
self._referral_url = QW.QLineEdit( self._referral_url_panel )
self._referral_url.setReadOnly( True )
#
self._api_url_panel = ClientGUICommon.StaticBox( self._options_panel, 'api url' )
self._api_lookup_converter = ClientGUIStringControls.StringConverterButton( self._api_url_panel, api_lookup_converter )
tt = 'This will let you generate an alternate URL for the client to use for the actual download whenever it encounters a URL in this class. You must have a separate URL class to match the API type (which will link to parsers).'
self._api_lookup_converter.setToolTip( tt )
self._api_url = QW.QLineEdit( self._api_url_panel )
self._api_url.setReadOnly( True )
#
self._next_gallery_page_panel = ClientGUICommon.StaticBox( self._options_panel, 'next gallery page' )
self._next_gallery_page_choice = ClientGUICommon.BetterChoice( self._next_gallery_page_panel )
self._next_gallery_page_delta = QP.MakeQSpinBox( self._next_gallery_page_panel, min=1, max=65536 )
self._next_gallery_page_url = QW.QLineEdit( self._next_gallery_page_panel )
self._next_gallery_page_url.setReadOnly( True )
#
headers_panel = ClientGUICommon.StaticBox( self._options_panel, 'header overrides' )
header_overrides = url_class.GetHeaderOverrides()
@ -996,11 +1080,7 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
#
self._next_gallery_page_panel = ClientGUICommon.StaticBox( self, 'next gallery page' )
self._next_gallery_page_choice = ClientGUICommon.BetterChoice( self._next_gallery_page_panel )
self._next_gallery_page_delta = QP.MakeQSpinBox( self._next_gallery_page_panel, min=1, max=65536 )
self._notebook.addTab( self._options_panel, 'options' )
#
@ -1017,40 +1097,6 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
self._normalised_url.setToolTip( tt )
( url_type, preferred_scheme, netloc, path_components, parameters, api_lookup_converter, send_referral_url, referral_url_converter, example_url ) = url_class.ToTuple()
self._send_referral_url = ClientGUICommon.BetterChoice( self )
for send_referral_url_type in ClientNetworkingDomain.SEND_REFERRAL_URL_TYPES:
self._send_referral_url.addItem( ClientNetworkingDomain.send_referral_url_string_lookup[ send_referral_url_type ], send_referral_url_type )
tt = 'Do not change this unless you know you need to. It fixes complicated problems.'
self._send_referral_url.setToolTip( tt )
self._referral_url_converter = ClientGUIStringControls.StringConverterButton( self, referral_url_converter )
tt = 'This will generate a referral URL from the original URL. If the URL needs a referral URL, and you can infer what that would be from just this URL, this will let hydrus download this URL without having to previously visit the referral URL (e.g. letting the user drag-and-drop import). It also lets you set up alternate referral URLs for perculiar situations.'
self._referral_url_converter.setToolTip( tt )
self._referral_url = QW.QLineEdit()
self._referral_url.setReadOnly( True )
self._api_lookup_converter = ClientGUIStringControls.StringConverterButton( self, api_lookup_converter )
tt = 'This will let you generate an alternate URL for the client to use for the actual download whenever it encounters a URL in this class. You must have a separate URL class to match the API type (which will link to parsers).'
self._api_lookup_converter.setToolTip( tt )
self._api_url = QW.QLineEdit( self )
self._api_url.setReadOnly( True )
self._next_gallery_page_url = QW.QLineEdit( self )
self._next_gallery_page_url.setReadOnly( True )
#
name = url_class.GetName()
@ -1074,7 +1120,7 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
self._path_components.AddDatas( path_components )
self._parameters.AddDatas( list(parameters.items()) )
self._parameters.AddDatas( list( parameters.items() ) )
self._parameters.Sort()
@ -1106,7 +1152,29 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
#
headers_panel.Add( self._header_overrides, CC.FLAGS_EXPAND_PERPENDICULAR )
headers_panel.Add( self._header_overrides, CC.FLAGS_EXPAND_BOTH_WAYS )
#
rows = []
rows.append( ( 'preferred scheme: ', self._preferred_scheme ) )
rows.append( ( 'network location: ', self._netloc ) )
rows.append( ( 'should subdomains also match this class?: ', self._match_subdomains ) )
gridbox_1 = ClientGUICommon.WrapInGrid( self._matching_panel, rows )
rows = []
rows.append( ( 'has single-value parameter(s): ', self._has_single_value_parameters ) )
rows.append( ( 'string match for single-value parameters: ', self._single_value_parameters_string_match ) )
gridbox_2 = ClientGUICommon.WrapInGrid( self._matching_panel, rows )
self._matching_panel.Add( gridbox_1, CC.FLAGS_EXPAND_PERPENDICULAR )
self._matching_panel.Add( path_components_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
self._matching_panel.Add( parameters_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
self._matching_panel.Add( gridbox_2, CC.FLAGS_EXPAND_PERPENDICULAR )
#
@ -1115,7 +1183,49 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
QP.AddToLayout( hbox, self._next_gallery_page_choice, CC.FLAGS_EXPAND_BOTH_WAYS )
QP.AddToLayout( hbox, self._next_gallery_page_delta, CC.FLAGS_CENTER_PERPENDICULAR )
rows = []
rows.append( ( 'next gallery page url: ', self._next_gallery_page_url ) )
gridbox = ClientGUICommon.WrapInGrid( self._next_gallery_page_panel, rows )
self._next_gallery_page_panel.Add( hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
self._next_gallery_page_panel.Add( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
rows = []
rows.append( ( 'send referral url?: ', self._send_referral_url ) )
rows.append( ( 'optional referral url converter: ', self._referral_url_converter ) )
rows.append( ( 'referral url: ', self._referral_url ) )
gridbox = ClientGUICommon.WrapInGrid( self._referral_url_panel, rows )
self._referral_url_panel.Add( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
rows = []
rows.append( ( 'optional api url converter: ', self._api_lookup_converter ) )
rows.append( ( 'api url: ', self._api_url ) )
gridbox = ClientGUICommon.WrapInGrid( self._api_url_panel, rows )
self._api_url_panel.Add( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
rows = []
rows.append( ( 'if matching by subdomain, keep it when normalising?: ', self._keep_matched_subdomains ) )
rows.append( ( 'alphabetise GET parameters when normalising?: ', self._alphabetise_get_parameters ) )
rows.append( ( 'keep fragment when normalising?: ', self._keep_fragment ) )
rows.append( ( 'post page can produce multiple files?: ', self._can_produce_multiple_files ) )
rows.append( ( 'associate a \'known url\' with resulting files?: ', self._should_be_associated_with_files ) )
gridbox = ClientGUICommon.WrapInGrid( self._options_panel, rows )
self._options_panel.Add( gridbox, CC.FLAGS_EXPAND_PERPENDICULAR )
self._options_panel.Add( self._api_url_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
self._options_panel.Add( self._referral_url_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
self._options_panel.Add( self._next_gallery_page_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
self._options_panel.Add( headers_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
#
@ -1123,37 +1233,20 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
rows.append( ( 'name: ', self._name ) )
rows.append( ( 'url type: ', self._url_type ) )
rows.append( ( 'preferred scheme: ', self._preferred_scheme ) )
rows.append( ( 'network location: ', self._netloc ) )
rows.append( ( 'should alphabetise GET parameters when normalising?: ', self._alphabetise_get_parameters ) )
rows.append( ( 'should subdomains also match this class?: ', self._match_subdomains ) )
rows.append( ( 'keep those subdomains when normalising?: ', self._keep_matched_subdomains ) )
rows.append( ( 'post page can produce multiple files?: ', self._can_produce_multiple_files ) )
rows.append( ( 'associate a \'known url\' with resulting files?: ', self._should_be_associated_with_files ) )
rows.append( ( 'keep fragment?: ', self._keep_fragment ) )
gridbox_1 = ClientGUICommon.WrapInGrid( self, rows )
gridbox_1 = ClientGUICommon.WrapInGrid( self._matching_panel, rows )
rows = []
rows.append( ( 'example url: ', self._example_url ) )
rows.append( ( 'normalised url: ', self._normalised_url ) )
rows.append( ( 'send referral url?: ', self._send_referral_url ) )
rows.append( ( 'optional referral url converter: ', self._referral_url_converter ) )
rows.append( ( 'referral url: ', self._referral_url ) )
rows.append( ( 'optional api url converter: ', self._api_lookup_converter ) )
rows.append( ( 'api url: ', self._api_url ) )
rows.append( ( 'next gallery page url: ', self._next_gallery_page_url ) )
gridbox_2 = ClientGUICommon.WrapInGrid( self, rows )
vbox = QP.VBoxLayout()
QP.AddToLayout( vbox, gridbox_1, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, path_components_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
QP.AddToLayout( vbox, parameters_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
QP.AddToLayout( vbox, headers_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._next_gallery_page_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._notebook, CC.FLAGS_EXPAND_BOTH_WAYS )
QP.AddToLayout( vbox, self._example_url_classes, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, gridbox_2, CC.FLAGS_EXPAND_PERPENDICULAR )
@ -1176,6 +1269,8 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
self._send_referral_url.currentIndexChanged.connect( self._UpdateControls )
self._referral_url_converter.valueChanged.connect( self._UpdateControls )
self._api_lookup_converter.valueChanged.connect( self._UpdateControls )
self._has_single_value_parameters.clicked.connect( self._UpdateControls )
self._single_value_parameters_string_match.valueChanged.connect( self._UpdateControls )
self._should_be_associated_with_files.clicked.connect( self.EventAssociationUpdate )
@ -1203,7 +1298,7 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
return
string_match = ClientParsing.StringMatch()
string_match = ClientStrings.StringMatch()
with ClientGUITopLevelWindowsPanels.DialogEdit( self, 'edit value' ) as dlg:
@ -1257,7 +1352,7 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
def _AddPathComponent( self ):
string_match = ClientParsing.StringMatch()
string_match = ClientStrings.StringMatch()
default = None
return self._EditPathComponent( ( string_match, default ) )
@ -1464,6 +1559,8 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
netloc = self._netloc.text()
path_components = self._path_components.GetData()
parameters = dict( self._parameters.GetData() )
has_single_value_parameters = self._has_single_value_parameters.isChecked()
single_value_parameters_string_match = self._single_value_parameters_string_match.GetValue()
header_overrides = self._header_overrides.GetValue()
api_lookup_converter = self._api_lookup_converter.GetValue()
send_referral_url = self._send_referral_url.GetValue()
@ -1482,6 +1579,8 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
netloc = netloc,
path_components = path_components,
parameters = parameters,
has_single_value_parameters = has_single_value_parameters,
single_value_parameters_string_match = single_value_parameters_string_match,
header_overrides = header_overrides,
api_lookup_converter = api_lookup_converter,
send_referral_url = send_referral_url,
@ -1568,6 +1667,8 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
self._next_gallery_page_panel.setEnabled( False )
self._single_value_parameters_string_match.setEnabled( self._has_single_value_parameters.isChecked() )
#
url_class = self._GetValue()

78
hydrus/client/gui/ClientGUIFileSeedCache.py
View File

@ -27,6 +27,7 @@ from hydrus.client.gui.lists import ClientGUIListCtrl
from hydrus.client.gui.widgets import ClientGUICommon
from hydrus.client.importing import ClientImportFileSeeds
from hydrus.client.importing.options import FileImportOptions
from hydrus.client.metadata import ClientTagSorting
def GetRetryIgnoredParam( window ):
@ -199,6 +200,83 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
ClientGUIMenus.AppendMenuItem( menu, 'copy sources', 'Copy all the selected sources to clipboard.', self._CopySelectedFileSeedData )
ClientGUIMenus.AppendMenuItem( menu, 'copy notes', 'Copy all the selected notes to clipboard.', self._CopySelectedNotes )
if len( selected_file_seeds ) == 1:
ClientGUIMenus.AppendSeparator( menu )
( selected_file_seed, ) = selected_file_seeds
hash_types_to_hashes = selected_file_seed.GetHashTypesToHashes()
if len( hash_types_to_hashes ) == 0:
ClientGUIMenus.AppendMenuLabel( menu, 'no hashes yet' )
else:
hash_submenu = QW.QMenu( menu )
for hash_type in ( 'sha256', 'md5', 'sha1', 'sha512' ):
if hash_type in hash_types_to_hashes:
h = hash_types_to_hashes[ hash_type ]
ClientGUIMenus.AppendMenuLabel( hash_submenu, '{}:{}'.format( hash_type, h.hex() ) )
ClientGUIMenus.AppendMenu( menu, hash_submenu, 'hashes' )
#
if selected_file_seed.IsURLFileImport():
urls = sorted( selected_file_seed.GetURLs() )
if len( urls ) == 0:
ClientGUIMenus.AppendMenuLabel( menu, 'no parsed urls' )
else:
url_submenu = QW.QMenu( menu )
for url in urls:
ClientGUIMenus.AppendMenuLabel( url_submenu, url )
ClientGUIMenus.AppendMenu( menu, url_submenu, 'parsed urls' )
#
tags = list( selected_file_seed.GetExternalTags() )
tag_sort = ClientTagSorting.TagSort( sort_type = ClientTagSorting.SORT_BY_HUMAN_TAG, sort_order = CC.SORT_ASC )
ClientTagSorting.SortTags( tag_sort, tags )
if len( tags ) == 0:
ClientGUIMenus.AppendMenuLabel( menu, 'no parsed tags' )
else:
tag_submenu = QW.QMenu( menu )
for tag in tags:
ClientGUIMenus.AppendMenuLabel( tag_submenu, tag )
ClientGUIMenus.AppendMenu( menu, tag_submenu, 'parsed tags' )
ClientGUIMenus.AppendSeparator( menu )
ClientGUIMenus.AppendMenuItem( menu, 'open sources', 'Open all the selected sources in your file explorer or web browser.', self._OpenSelectedFileSeedData )

14
hydrus/client/gui/ClientGUIMPV.py
View File

@ -67,6 +67,10 @@ def GetClientAPIVersionString():
'''
def log_handler( loglevel, component, message ):
HydrusData.DebugPrint( '[{}] {}: {}'.format( loglevel, component, message ) )
#Not sure how well this works with hardware acceleration. This just renders to a QWidget. In my tests it seems fine, even with vdpau video out, but I'm not 100% sure it actually uses hardware acceleration.
#Here is an example on how to render into a QOpenGLWidget instead: https://gist.github.com/cosven/b313de2acce1b7e15afda263779c0afc
class mpvWidget( QW.QWidget ):
@ -88,8 +92,10 @@ class mpvWidget( QW.QWidget ):
self.setAttribute( QC.Qt.WA_DontCreateNativeAncestors )
self.setAttribute( QC.Qt.WA_NativeWindow )
loglevel = 'debug' if HG.mpv_report_mode else 'fatal'
# loglevels: fatal, error, debug
self._player = mpv.MPV( wid = str( int( self.winId() ) ), log_handler = print, loglevel = 'fatal' )
self._player = mpv.MPV( wid = str( int( self.winId() ) ), log_handler = log_handler, loglevel = loglevel )
# hydev notes on OSC:
# OSC is by default off, default input bindings are by default off
@ -130,6 +136,7 @@ class mpvWidget( QW.QWidget ):
HG.client_controller.sub( self, 'UpdateAudioMute', 'new_audio_mute' )
HG.client_controller.sub( self, 'UpdateAudioVolume', 'new_audio_volume' )
HG.client_controller.sub( self, 'UpdateConf', 'notify_new_options' )
HG.client_controller.sub( self, 'SetLogLevel', 'set_mpv_log_level' )
self._my_shortcut_handler = ClientGUIShortcuts.ShortcutsHandler( self, [], catch_mouse = True )
@ -456,6 +463,11 @@ class mpvWidget( QW.QWidget ):
self._my_shortcut_handler.SetShortcuts( [ shortcut_set ] )
def SetLogLevel( self, level: str ):
self._player.set_loglevel( level )
def SetMedia( self, media, start_paused = False ):
if media == self._media:

11
hydrus/client/gui/ClientGUIParsing.py
View File

@ -24,6 +24,7 @@ from hydrus.client import ClientDefaults
from hydrus.client import ClientParsing
from hydrus.client import ClientPaths
from hydrus.client import ClientSerialisable
from hydrus.client import ClientStrings
from hydrus.client import ClientThreading
from hydrus.client.gui import ClientGUIDialogs
from hydrus.client.gui import ClientGUIDialogsQuick
@ -1444,7 +1445,7 @@ class EditJSONParsingRulePanel( ClientGUIScrolledPanels.EditPanel ):
self._parse_rule_type.addItem( 'all dictionary/list items', ClientParsing.JSON_PARSE_RULE_TYPE_ALL_ITEMS )
self._parse_rule_type.addItem( 'indexed item', ClientParsing.JSON_PARSE_RULE_TYPE_INDEXED_ITEM )
string_match = ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = 'posts', example_string = 'posts' )
string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'posts', example_string = 'posts' )
self._string_match = ClientGUIStringPanels.EditStringMatchPanel( self, string_match )
@ -1651,7 +1652,7 @@ class EditJSONFormulaPanel( ClientGUIScrolledPanels.EditPanel ):
with ClientGUITopLevelWindowsPanels.DialogEdit( self, dlg_title, frame_key = 'deeply_nested_dialog' ) as dlg:
new_rule = ( ClientParsing.JSON_PARSE_RULE_TYPE_DICT_KEY, ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = 'posts', example_string = 'posts' ) )
new_rule = ( ClientParsing.JSON_PARSE_RULE_TYPE_DICT_KEY, ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'posts', example_string = 'posts' ) )
panel = EditJSONParsingRulePanel( dlg, new_rule )
@ -1858,7 +1859,7 @@ class EditContentParserPanel( ClientGUIScrolledPanels.EditPanel ):
self._veto_panel = QW.QWidget( self._content_panel )
self._veto_if_matches_found = QW.QCheckBox( self._veto_panel )
self._string_match = ClientGUIStringPanels.EditStringMatchPanel( self._veto_panel, ClientParsing.StringMatch() )
self._string_match = ClientGUIStringPanels.EditStringMatchPanel( self._veto_panel, ClientStrings.StringMatch() )
self._temp_variable_panel = QW.QWidget( self._content_panel )
@ -3853,7 +3854,7 @@ class ManageParsingScriptsPanel( ClientGUIScrolledPanels.ManagePanel ):
url = ''
query_type = HC.GET
file_identifier_type = ClientParsing.FILE_IDENTIFIER_TYPE_MD5
file_identifier_string_converter = ClientParsing.StringConverter( ( ( ClientParsing.STRING_CONVERSION_ENCODE, 'hex' ), ), 'some hash bytes' )
file_identifier_string_converter = ClientStrings.StringConverter( ( ( ClientStrings.STRING_CONVERSION_ENCODE, 'hex' ), ), 'some hash bytes' )
file_identifier_arg_name = 'md5'
static_args = {}
children = []
@ -4611,7 +4612,7 @@ class TestPanelFormula( TestPanel ):
try:
formula.SetStringProcessor( ClientParsing.StringProcessor() )
formula.SetStringProcessor( ClientStrings.StringProcessor() )
texts = formula.Parse( example_parsing_context, self._example_data_raw )

37
hydrus/client/gui/ClientGUIStringControls.py
View File

@ -8,6 +8,7 @@ from hydrus.core import HydrusText
from hydrus.client import ClientConstants as CC
from hydrus.client import ClientParsing
from hydrus.client import ClientStrings
from hydrus.client.gui import ClientGUIDialogs
from hydrus.client.gui import ClientGUIScrolledPanels
from hydrus.client.gui import ClientGUIStringPanels
@ -21,7 +22,7 @@ class StringConverterButton( ClientGUICommon.BetterButton ):
valueChanged = QC.Signal()
def __init__( self, parent, string_converter: ClientParsing.StringConverter ):
def __init__( self, parent, string_converter: ClientStrings.StringConverter ):
ClientGUICommon.BetterButton.__init__( self, parent, 'edit string converter', self._Edit )
@ -62,7 +63,7 @@ class StringConverterButton( ClientGUICommon.BetterButton ):
self.setText( elided_label )
def GetValue( self ) -> ClientParsing.StringConverter:
def GetValue( self ) -> ClientStrings.StringConverter:
return self._string_converter
@ -72,7 +73,7 @@ class StringConverterButton( ClientGUICommon.BetterButton ):
self._example_string_override = example_string
def SetValue( self, string_converter: ClientParsing.StringConverter ):
def SetValue( self, string_converter: ClientStrings.StringConverter ):
self._string_converter = string_converter
@ -85,7 +86,7 @@ class StringMatchButton( ClientGUICommon.BetterButton ):
valueChanged = QC.Signal()
def __init__( self, parent, string_match: ClientParsing.StringMatch ):
def __init__( self, parent, string_match: ClientStrings.StringMatch ):
ClientGUICommon.BetterButton.__init__( self, parent, 'edit string match', self._Edit )
@ -108,6 +109,8 @@ class StringMatchButton( ClientGUICommon.BetterButton ):
self._UpdateLabel()
self.valueChanged.emit()
@ -118,12 +121,12 @@ class StringMatchButton( ClientGUICommon.BetterButton ):
self.setText( label )
def GetValue( self ) -> ClientParsing.StringMatch:
def GetValue( self ) -> ClientStrings.StringMatch:
return self._string_match
def SetValue( self, string_match: ClientParsing.StringMatch ):
def SetValue( self, string_match: ClientStrings.StringMatch ):
self._string_match = string_match
@ -132,7 +135,7 @@ class StringMatchButton( ClientGUICommon.BetterButton ):
class StringProcessorButton( ClientGUICommon.BetterButton ):
def __init__( self, parent, string_processor: ClientParsing.StringProcessor, test_data_callable: typing.Callable[ [], ClientParsing.ParsingTestData ] ):
def __init__( self, parent, string_processor: ClientStrings.StringProcessor, test_data_callable: typing.Callable[ [], ClientParsing.ParsingTestData ] ):
ClientGUICommon.BetterButton.__init__( self, parent, 'edit string processor', self._Edit )
@ -179,12 +182,12 @@ class StringProcessorButton( ClientGUICommon.BetterButton ):
self.setText( label )
def GetValue( self ) -> ClientParsing.StringProcessor:
def GetValue( self ) -> ClientStrings.StringProcessor:
return self._string_processor
def SetValue( self, string_processor: ClientParsing.StringProcessor ):
def SetValue( self, string_processor: ClientStrings.StringProcessor ):
self._string_processor = string_processor
@ -193,7 +196,7 @@ class StringProcessorButton( ClientGUICommon.BetterButton ):
class StringMatchToStringMatchDictControl( QW.QWidget ):
def __init__( self, parent, initial_dict: typing.Dict[ ClientParsing.StringMatch, ClientParsing.StringMatch ], min_height = 10, key_name = 'key' ):
def __init__( self, parent, initial_dict: typing.Dict[ ClientStrings.StringMatch, ClientStrings.StringMatch ], min_height = 10, key_name = 'key' ):
QW.QWidget.__init__( self, parent )
@ -243,7 +246,7 @@ class StringMatchToStringMatchDictControl( QW.QWidget ):
with ClientGUITopLevelWindowsPanels.DialogEdit( self, 'edit ' + self._key_name ) as dlg:
string_match = ClientParsing.StringMatch()
string_match = ClientStrings.StringMatch()
panel = ClientGUIStringPanels.EditStringMatchPanel( dlg, string_match )
@ -261,7 +264,7 @@ class StringMatchToStringMatchDictControl( QW.QWidget ):
with ClientGUITopLevelWindowsPanels.DialogEdit( self, 'edit match' ) as dlg:
string_match = ClientParsing.StringMatch()
string_match = ClientStrings.StringMatch()
panel = ClientGUIStringPanels.EditStringMatchPanel( dlg, string_match )
@ -326,7 +329,7 @@ class StringMatchToStringMatchDictControl( QW.QWidget ):
self._listctrl.Sort()
def GetValue( self ) -> typing.Dict[ str, ClientParsing.StringMatch ]:
def GetValue( self ) -> typing.Dict[ str, ClientStrings.StringMatch ]:
value_dict = dict( self._listctrl.GetData() )
@ -530,7 +533,7 @@ class StringToStringDictControl( QW.QWidget ):
class StringToStringMatchDictControl( QW.QWidget ):
def __init__( self, parent, initial_dict: typing.Dict[ str, ClientParsing.StringMatch ], min_height = 10, key_name = 'key' ):
def __init__( self, parent, initial_dict: typing.Dict[ str, ClientStrings.StringMatch ], min_height = 10, key_name = 'key' ):
QW.QWidget.__init__( self, parent )
@ -592,7 +595,7 @@ class StringToStringMatchDictControl( QW.QWidget ):
with ClientGUITopLevelWindowsPanels.DialogEdit( self, 'edit match' ) as dlg:
string_match = ClientParsing.StringMatch()
string_match = ClientStrings.StringMatch()
panel = ClientGUIStringPanels.EditStringMatchPanel( dlg, string_match )
@ -638,7 +641,7 @@ class StringToStringMatchDictControl( QW.QWidget ):
with ClientGUITopLevelWindowsPanels.DialogEdit( self, 'edit match' ) as dlg:
string_match = ClientParsing.StringMatch()
string_match = ClientStrings.StringMatch()
panel = ClientGUIStringPanels.EditStringMatchPanel( dlg, string_match )
@ -669,7 +672,7 @@ class StringToStringMatchDictControl( QW.QWidget ):
return { key for ( key, value ) in self._listctrl.GetData() }
def GetValue( self ) -> typing.Dict[ str, ClientParsing.StringMatch ]:
def GetValue( self ) -> typing.Dict[ str, ClientStrings.StringMatch ]:
value_dict = dict( self._listctrl.GetData() )

193
hydrus/client/gui/ClientGUIStringPanels.py
View File

@ -10,6 +10,7 @@ from hydrus.core import HydrusExceptions
from hydrus.client import ClientConstants as CC
from hydrus.client import ClientParsing
from hydrus.client import ClientStrings
from hydrus.client.gui import ClientGUIDialogsQuick
from hydrus.client.gui import ClientGUIFunctions
from hydrus.client.gui import ClientGUIScrolledPanels
@ -26,7 +27,7 @@ class MultilineStringConversionTestPanel( QW.QWidget ):
textSelected = QC.Signal( str )
def __init__( self, parent: QW.QWidget, string_processor: ClientParsing.StringProcessor ):
def __init__( self, parent: QW.QWidget, string_processor: ClientStrings.StringProcessor ):
QW.QWidget.__init__( self, parent )
@ -121,7 +122,7 @@ class MultilineStringConversionTestPanel( QW.QWidget ):
return results
def SetStringProcessor( self, string_processor: ClientParsing.StringProcessor ):
def SetStringProcessor( self, string_processor: ClientStrings.StringProcessor ):
self._string_processor = string_processor
@ -155,7 +156,7 @@ class MultilineStringConversionTestPanel( QW.QWidget ):
class SingleStringConversionTestPanel( QW.QWidget ):
def __init__( self, parent: QW.QWidget, string_processor: ClientParsing.StringProcessor ):
def __init__( self, parent: QW.QWidget, string_processor: ClientStrings.StringProcessor ):
QW.QWidget.__init__( self, parent )
@ -193,7 +194,7 @@ class SingleStringConversionTestPanel( QW.QWidget ):
for i in range( len( processing_steps ) ):
if isinstance( processing_steps[i], ClientParsing.StringSlicer ):
if isinstance( processing_steps[i], ClientStrings.StringSlicer ):
continue
@ -276,11 +277,11 @@ class SingleStringConversionTestPanel( QW.QWidget ):
return self._example_string.text()
def SetStringProcessor( self, string_processor: ClientParsing.StringProcessor ):
def SetStringProcessor( self, string_processor: ClientStrings.StringProcessor ):
self._string_processor = string_processor
if True in ( isinstance( processing_step, ClientParsing.StringSlicer ) for processing_step in self._string_processor.GetProcessingSteps() ):
if True in ( isinstance( processing_step, ClientStrings.StringSlicer ) for processing_step in self._string_processor.GetProcessingSteps() ):
self.setToolTip( 'String Slicing is ignored here.' )
@ -301,7 +302,7 @@ class SingleStringConversionTestPanel( QW.QWidget ):
class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent: QW.QWidget, string_converter: ClientParsing.StringConverter, example_string_override = None ):
def __init__( self, parent: QW.QWidget, string_converter: ClientStrings.StringConverter, example_string_override = None ):
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
@ -361,7 +362,7 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
def _AddConversion( self ):
conversion_type = ClientParsing.STRING_CONVERSION_APPEND_TEXT
conversion_type = ClientStrings.STRING_CONVERSION_APPEND_TEXT
data = 'extra text'
try:
@ -437,7 +438,7 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
( number, conversion_type, data ) = conversion
pretty_number = HydrusData.ToHumanInt( number )
pretty_conversion = ClientParsing.StringConverter.ConversionToString( ( conversion_type, data ) )
pretty_conversion = ClientStrings.StringConverter.ConversionToString( ( conversion_type, data ) )
string_converter = self._GetValue()
@ -578,7 +579,7 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
example_string = self._example_string.text()
string_converter = ClientParsing.StringConverter( conversions, example_string )
string_converter = ClientStrings.StringConverter( conversions, example_string )
return string_converter
@ -672,9 +673,9 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
self._conversion_type = ClientGUICommon.BetterChoice( self._control_panel )
for t_type in ( ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_END, ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_END, ClientParsing.STRING_CONVERSION_PREPEND_TEXT, ClientParsing.STRING_CONVERSION_APPEND_TEXT, ClientParsing.STRING_CONVERSION_ENCODE, ClientParsing.STRING_CONVERSION_DECODE, ClientParsing.STRING_CONVERSION_REVERSE, ClientParsing.STRING_CONVERSION_REGEX_SUB, ClientParsing.STRING_CONVERSION_DATE_DECODE, ClientParsing.STRING_CONVERSION_DATE_ENCODE, ClientParsing.STRING_CONVERSION_INTEGER_ADDITION ):
for t_type in ( ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_END, ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_END, ClientStrings.STRING_CONVERSION_PREPEND_TEXT, ClientStrings.STRING_CONVERSION_APPEND_TEXT, ClientStrings.STRING_CONVERSION_ENCODE, ClientStrings.STRING_CONVERSION_DECODE, ClientStrings.STRING_CONVERSION_REVERSE, ClientStrings.STRING_CONVERSION_REGEX_SUB, ClientStrings.STRING_CONVERSION_DATE_DECODE, ClientStrings.STRING_CONVERSION_DATE_ENCODE, ClientStrings.STRING_CONVERSION_INTEGER_ADDITION ):
self._conversion_type.addItem( ClientParsing.conversion_type_str_lookup[ t_type ], t_type )
self._conversion_type.addItem( ClientStrings.conversion_type_str_lookup[ t_type ], t_type )
self._data_text = QW.QLineEdit( self._control_panel )
@ -738,22 +739,22 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
#
if conversion_type == ClientParsing.STRING_CONVERSION_ENCODE:
if conversion_type == ClientStrings.STRING_CONVERSION_ENCODE:
self._data_encoding.SetValue( data )
elif conversion_type == ClientParsing.STRING_CONVERSION_DECODE:
elif conversion_type == ClientStrings.STRING_CONVERSION_DECODE:
self._data_decoding.SetValue( data )
elif conversion_type == ClientParsing.STRING_CONVERSION_REGEX_SUB:
elif conversion_type == ClientStrings.STRING_CONVERSION_REGEX_SUB:
( pattern, repl ) = data
self._data_text.setText( pattern )
self._data_regex_repl.setText( repl )
elif conversion_type == ClientParsing.STRING_CONVERSION_DATE_DECODE:
elif conversion_type == ClientStrings.STRING_CONVERSION_DATE_DECODE:
( phrase, timezone_type, timezone_offset ) = data
@ -761,7 +762,7 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
self._data_timezone_decode.SetValue( timezone_type )
self._data_timezone_offset.setValue( timezone_offset )
elif conversion_type == ClientParsing.STRING_CONVERSION_DATE_ENCODE:
elif conversion_type == ClientStrings.STRING_CONVERSION_DATE_ENCODE:
( phrase, timezone_type ) = data
@ -878,37 +879,37 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
conversion_type = self._conversion_type.GetValue()
if conversion_type == ClientParsing.STRING_CONVERSION_ENCODE:
if conversion_type == ClientStrings.STRING_CONVERSION_ENCODE:
self._data_encoding_label.setVisible( True )
self._data_encoding.setVisible( True )
elif conversion_type == ClientParsing.STRING_CONVERSION_DECODE:
elif conversion_type == ClientStrings.STRING_CONVERSION_DECODE:
self._data_decoding_label.setVisible( True )
self._data_decoding.setVisible( True )
elif conversion_type in ( ClientParsing.STRING_CONVERSION_PREPEND_TEXT, ClientParsing.STRING_CONVERSION_APPEND_TEXT, ClientParsing.STRING_CONVERSION_DATE_DECODE, ClientParsing.STRING_CONVERSION_DATE_ENCODE, ClientParsing.STRING_CONVERSION_REGEX_SUB ):
elif conversion_type in ( ClientStrings.STRING_CONVERSION_PREPEND_TEXT, ClientStrings.STRING_CONVERSION_APPEND_TEXT, ClientStrings.STRING_CONVERSION_DATE_DECODE, ClientStrings.STRING_CONVERSION_DATE_ENCODE, ClientStrings.STRING_CONVERSION_REGEX_SUB ):
self._data_text_label.setVisible( True )
self._data_text.setVisible( True )
data_text_label = 'string data: '
if conversion_type == ClientParsing.STRING_CONVERSION_PREPEND_TEXT:
if conversion_type == ClientStrings.STRING_CONVERSION_PREPEND_TEXT:
data_text_label = 'text to prepend: '
elif conversion_type == ClientParsing.STRING_CONVERSION_APPEND_TEXT:
elif conversion_type == ClientStrings.STRING_CONVERSION_APPEND_TEXT:
data_text_label = 'text to append: '
elif conversion_type in ( ClientParsing.STRING_CONVERSION_DATE_DECODE, ClientParsing.STRING_CONVERSION_DATE_ENCODE ):
elif conversion_type in ( ClientStrings.STRING_CONVERSION_DATE_DECODE, ClientStrings.STRING_CONVERSION_DATE_ENCODE ):
self._data_date_link_label.setVisible( True )
self._data_date_link.setVisible( True )
if conversion_type == ClientParsing.STRING_CONVERSION_DATE_DECODE:
if conversion_type == ClientStrings.STRING_CONVERSION_DATE_DECODE:
data_text_label = 'date decode phrase: '
@ -921,7 +922,7 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
self._data_timezone_offset.setVisible( True )
elif conversion_type == ClientParsing.STRING_CONVERSION_DATE_ENCODE:
elif conversion_type == ClientStrings.STRING_CONVERSION_DATE_ENCODE:
data_text_label = 'date encode phrase: '
@ -929,7 +930,7 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
self._data_timezone_encode.setVisible( True )
elif conversion_type == ClientParsing.STRING_CONVERSION_REGEX_SUB:
elif conversion_type == ClientStrings.STRING_CONVERSION_REGEX_SUB:
data_text_label = 'regex pattern: '
@ -939,12 +940,12 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
self._data_text_label.setText( data_text_label )
elif conversion_type in ( ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_END, ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_END, ClientParsing.STRING_CONVERSION_INTEGER_ADDITION ):
elif conversion_type in ( ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_END, ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_END, ClientStrings.STRING_CONVERSION_INTEGER_ADDITION ):
self._data_number_label.setVisible( True )
self._data_number.setVisible( True )
if conversion_type == ClientParsing.STRING_CONVERSION_INTEGER_ADDITION:
if conversion_type == ClientStrings.STRING_CONVERSION_INTEGER_ADDITION:
self._data_number.setMinimum( -65535 )
@ -955,23 +956,23 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
data_number_label = 'number data: '
if conversion_type == ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING:
if conversion_type == ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING:
data_number_label = 'characters to remove: '
elif conversion_type == ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_END:
elif conversion_type == ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_END:
data_number_label = 'characters to remove: '
elif conversion_type == ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING:
elif conversion_type == ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING:
data_number_label = 'characters to take: '
elif conversion_type == ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_END:
elif conversion_type == ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_END:
data_number_label = 'characters to take: '
elif conversion_type == ClientParsing.STRING_CONVERSION_INTEGER_ADDITION:
elif conversion_type == ClientStrings.STRING_CONVERSION_INTEGER_ADDITION:
data_number_label = 'number to add: '
@ -986,7 +987,7 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
conversions = [ self.GetValue() ]
string_converter = ClientParsing.StringConverter( conversions, self._example_text )
string_converter = ClientStrings.StringConverter( conversions, self._example_text )
example_conversion = string_converter.Convert( self._example_text )
@ -1009,30 +1010,30 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
conversion_type = self._conversion_type.GetValue()
if conversion_type == ClientParsing.STRING_CONVERSION_ENCODE:
if conversion_type == ClientStrings.STRING_CONVERSION_ENCODE:
data = self._data_encoding.GetValue()
elif conversion_type == ClientParsing.STRING_CONVERSION_DECODE:
elif conversion_type == ClientStrings.STRING_CONVERSION_DECODE:
data = self._data_decoding.GetValue()
elif conversion_type in ( ClientParsing.STRING_CONVERSION_PREPEND_TEXT, ClientParsing.STRING_CONVERSION_APPEND_TEXT ):
elif conversion_type in ( ClientStrings.STRING_CONVERSION_PREPEND_TEXT, ClientStrings.STRING_CONVERSION_APPEND_TEXT ):
data = self._data_text.text()
elif conversion_type in ( ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_END, ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_END, ClientParsing.STRING_CONVERSION_INTEGER_ADDITION ):
elif conversion_type in ( ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_END, ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_END, ClientStrings.STRING_CONVERSION_INTEGER_ADDITION ):
data = self._data_number.value()
elif conversion_type == ClientParsing.STRING_CONVERSION_REGEX_SUB:
elif conversion_type == ClientStrings.STRING_CONVERSION_REGEX_SUB:
pattern = self._data_text.text()
repl = self._data_regex_repl.text()
data = ( pattern, repl )
elif conversion_type == ClientParsing.STRING_CONVERSION_DATE_DECODE:
elif conversion_type == ClientStrings.STRING_CONVERSION_DATE_DECODE:
phrase = self._data_text.text()
timezone_time = self._data_timezone_decode.GetValue()
@ -1040,7 +1041,7 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
data = ( phrase, timezone_time, timezone_offset )
elif conversion_type == ClientParsing.STRING_CONVERSION_DATE_ENCODE:
elif conversion_type == ClientStrings.STRING_CONVERSION_DATE_ENCODE:
phrase = self._data_text.text()
timezone_time = self._data_timezone_encode.GetValue()
@ -1058,25 +1059,25 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
class EditStringMatchPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent: QW.QWidget, string_match: ClientParsing.StringMatch, test_data = typing.Optional[ ClientParsing.ParsingTestData ] ):
def __init__( self, parent: QW.QWidget, string_match: ClientStrings.StringMatch, test_data = typing.Optional[ ClientParsing.ParsingTestData ] ):
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
self._match_type = ClientGUICommon.BetterChoice( self )
self._match_type.addItem( 'any characters', ClientParsing.STRING_MATCH_ANY )
self._match_type.addItem( 'fixed characters', ClientParsing.STRING_MATCH_FIXED )
self._match_type.addItem( 'character set', ClientParsing.STRING_MATCH_FLEXIBLE )
self._match_type.addItem( 'regex', ClientParsing.STRING_MATCH_REGEX )
self._match_type.addItem( 'any characters', ClientStrings.STRING_MATCH_ANY )
self._match_type.addItem( 'fixed characters', ClientStrings.STRING_MATCH_FIXED )
self._match_type.addItem( 'character set', ClientStrings.STRING_MATCH_FLEXIBLE )
self._match_type.addItem( 'regex', ClientStrings.STRING_MATCH_REGEX )
self._match_value_fixed_input = QW.QLineEdit( self )
self._match_value_regex_input = QW.QLineEdit( self )
self._match_value_flexible_input = ClientGUICommon.BetterChoice( self )
self._match_value_flexible_input.addItem( 'alphabetic characters (a-zA-Z)', ClientParsing.ALPHA )
self._match_value_flexible_input.addItem( 'alphanumeric characters (a-zA-Z0-9)', ClientParsing.ALPHANUMERIC )
self._match_value_flexible_input.addItem( 'numeric characters (0-9)', ClientParsing.NUMERIC )
self._match_value_flexible_input.addItem( 'alphabetic characters (a-zA-Z)', ClientStrings.ALPHA )
self._match_value_flexible_input.addItem( 'alphanumeric characters (a-zA-Z0-9)', ClientStrings.ALPHANUMERIC )
self._match_value_flexible_input.addItem( 'numeric characters (0-9)', ClientStrings.NUMERIC )
self._min_chars = ClientGUICommon.NoneableSpinCtrl( self, min = 1, max = 65535, unit = 'characters', none_phrase = 'no limit' )
self._max_chars = ClientGUICommon.NoneableSpinCtrl( self, min = 1, max = 65535, unit = 'characters', none_phrase = 'no limit' )
@ -1132,24 +1133,24 @@ class EditStringMatchPanel( ClientGUIScrolledPanels.EditPanel ):
match_type = self._match_type.GetValue()
if match_type == ClientParsing.STRING_MATCH_ANY:
if match_type == ClientStrings.STRING_MATCH_ANY:
match_value = ''
elif match_type == ClientParsing.STRING_MATCH_FLEXIBLE:
elif match_type == ClientStrings.STRING_MATCH_FLEXIBLE:
match_value = self._match_value_flexible_input.GetValue()
elif match_type == ClientParsing.STRING_MATCH_FIXED:
elif match_type == ClientStrings.STRING_MATCH_FIXED:
match_value = self._match_value_fixed_input.text()
elif match_type == ClientParsing.STRING_MATCH_REGEX:
elif match_type == ClientStrings.STRING_MATCH_REGEX:
match_value = self._match_value_regex_input.text()
if match_type == ClientParsing.STRING_MATCH_FIXED:
if match_type == ClientStrings.STRING_MATCH_FIXED:
min_chars = None
max_chars = None
@ -1162,7 +1163,7 @@ class EditStringMatchPanel( ClientGUIScrolledPanels.EditPanel ):
example_string = self._example_string.text()
string_match = ClientParsing.StringMatch( match_type = match_type, match_value = match_value, min_chars = min_chars, max_chars = max_chars, example_string = example_string )
string_match = ClientStrings.StringMatch( match_type = match_type, match_value = match_value, min_chars = min_chars, max_chars = max_chars, example_string = example_string )
return string_match
@ -1185,7 +1186,7 @@ class EditStringMatchPanel( ClientGUIScrolledPanels.EditPanel ):
self._max_chars.setVisible( False )
self._example_string.setVisible( False )
if match_type == ClientParsing.STRING_MATCH_FIXED:
if match_type == ClientStrings.STRING_MATCH_FIXED:
self._match_value_fixed_input_label.setVisible( True )
self._match_value_fixed_input.setVisible( True )
@ -1200,12 +1201,12 @@ class EditStringMatchPanel( ClientGUIScrolledPanels.EditPanel ):
self._max_chars.setVisible( True )
self._example_string.setVisible( True )
if match_type == ClientParsing.STRING_MATCH_FLEXIBLE:
if match_type == ClientStrings.STRING_MATCH_FLEXIBLE:
self._match_value_flexible_input_label.setVisible( True )
self._match_value_flexible_input.setVisible( True )
elif match_type == ClientParsing.STRING_MATCH_REGEX:
elif match_type == ClientStrings.STRING_MATCH_REGEX:
self._match_value_regex_input_label.setVisible( True )
self._match_value_regex_input.setVisible( True )
@ -1219,7 +1220,7 @@ class EditStringMatchPanel( ClientGUIScrolledPanels.EditPanel ):
match_type = self._match_type.GetValue()
if match_type == ClientParsing.STRING_MATCH_FIXED:
if match_type == ClientStrings.STRING_MATCH_FIXED:
self._example_string_matches.clear()
@ -1262,23 +1263,23 @@ class EditStringMatchPanel( ClientGUIScrolledPanels.EditPanel ):
return string_match
def SetValue( self, string_match: ClientParsing.StringMatch ):
def SetValue( self, string_match: ClientStrings.StringMatch ):
( match_type, match_value, min_chars, max_chars, example_string ) = string_match.ToTuple()
self._match_type.SetValue( match_type )
self._match_value_flexible_input.SetValue( ClientParsing.ALPHA )
self._match_value_flexible_input.SetValue( ClientStrings.ALPHA )
if match_type == ClientParsing.STRING_MATCH_FIXED:
if match_type == ClientStrings.STRING_MATCH_FIXED:
self._match_value_fixed_input.setText( match_value )
elif match_type == ClientParsing.STRING_MATCH_FLEXIBLE:
elif match_type == ClientStrings.STRING_MATCH_FLEXIBLE:
self._match_value_flexible_input.SetValue( match_value )
elif match_type == ClientParsing.STRING_MATCH_REGEX:
elif match_type == ClientStrings.STRING_MATCH_REGEX:
self._match_value_regex_input.setText( match_value )
@ -1296,7 +1297,7 @@ SELECT_RANGE = 1
class EditStringSlicerPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent, string_slicer: ClientParsing.StringSlicer, test_data: typing.Sequence[ str ] = [] ):
def __init__( self, parent, string_slicer: ClientStrings.StringSlicer, test_data: typing.Sequence[ str ] = [] ):
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
@ -1416,7 +1417,7 @@ class EditStringSlicerPanel( ClientGUIScrolledPanels.EditPanel ):
index_end = self._index_end.GetValue()
string_slicer = ClientParsing.StringSlicer( index_start = index_start, index_end = index_end )
string_slicer = ClientStrings.StringSlicer( index_start = index_start, index_end = index_end )
return string_slicer
@ -1463,7 +1464,7 @@ class EditStringSlicerPanel( ClientGUIScrolledPanels.EditPanel ):
return string_slicer
def SetValue( self, string_slicer: ClientParsing.StringSlicer ):
def SetValue( self, string_slicer: ClientStrings.StringSlicer ):
( index_start, index_end ) = string_slicer.GetIndexStartEnd()
@ -1486,7 +1487,7 @@ class EditStringSlicerPanel( ClientGUIScrolledPanels.EditPanel ):
class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent, string_sorter: ClientParsing.StringSorter, test_data: typing.Sequence[ str ] = [] ):
def __init__( self, parent, string_sorter: ClientStrings.StringSorter, test_data: typing.Sequence[ str ] = [] ):
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
@ -1496,9 +1497,9 @@ class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
self._sort_type = ClientGUICommon.BetterChoice( self._controls_panel )
self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT ], ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT )
self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC ], ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC )
self._sort_type.addItem( ClientParsing.sort_str_enum[ ClientParsing.CONTENT_PARSER_SORT_TYPE_REVERSE ], ClientParsing.CONTENT_PARSER_SORT_TYPE_REVERSE )
self._sort_type.addItem( ClientStrings.sort_str_enum[ ClientStrings.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT ], ClientStrings.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT )
self._sort_type.addItem( ClientStrings.sort_str_enum[ ClientStrings.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC ], ClientStrings.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC )
self._sort_type.addItem( ClientStrings.sort_str_enum[ ClientStrings.CONTENT_PARSER_SORT_TYPE_REVERSE ], ClientStrings.CONTENT_PARSER_SORT_TYPE_REVERSE )
tt = 'Human sort sorts numbers as you understand them. "image 2" comes before "image 10". Lexicographic compares each character in turn. "image 02" comes before "image 10", which comes before "image 2".'
@ -1568,7 +1569,7 @@ class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
asc = self._asc.isChecked()
regex = self._regex.GetValue()
string_sorter = ClientParsing.StringSorter( sort_type = sort_type, asc = asc, regex = regex )
string_sorter = ClientStrings.StringSorter( sort_type = sort_type, asc = asc, regex = regex )
return string_sorter
@ -1626,7 +1627,7 @@ class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
return string_sorter
def SetValue( self, string_sorter: ClientParsing.StringSorter ):
def SetValue( self, string_sorter: ClientStrings.StringSorter ):
sort_type = string_sorter.GetSortType()
asc = string_sorter.GetAscending()
@ -1641,7 +1642,7 @@ class EditStringSorterPanel( ClientGUIScrolledPanels.EditPanel ):
class EditStringSplitterPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent, string_splitter: ClientParsing.StringSplitter, example_string: str = '' ):
def __init__( self, parent, string_splitter: ClientStrings.StringSplitter, example_string: str = '' ):
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
@ -1707,7 +1708,7 @@ class EditStringSplitterPanel( ClientGUIScrolledPanels.EditPanel ):
separator = self._separator.text()
max_splits = self._max_splits.GetValue()
string_splitter = ClientParsing.StringSplitter( separator = separator, max_splits = max_splits )
string_splitter = ClientStrings.StringSplitter( separator = separator, max_splits = max_splits )
return string_splitter
@ -1733,7 +1734,7 @@ class EditStringSplitterPanel( ClientGUIScrolledPanels.EditPanel ):
return string_splitter
def SetValue( self, string_splitter: ClientParsing.StringSplitter ):
def SetValue( self, string_splitter: ClientStrings.StringSplitter ):
separator = string_splitter.GetSeparator()
max_splits = string_splitter.GetMaxSplits()
@ -1746,7 +1747,7 @@ class EditStringSplitterPanel( ClientGUIScrolledPanels.EditPanel ):
class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent, string_processor: ClientParsing.StringProcessor, test_data: ClientParsing.ParsingTestData ):
def __init__( self, parent, string_processor: ClientStrings.StringProcessor, test_data: ClientParsing.ParsingTestData ):
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
@ -1804,11 +1805,11 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
def _Add( self ):
choice_tuples = [
( 'String Match', ClientParsing.StringMatch, 'An object that filters strings.' ),
( 'String Converter', ClientParsing.StringConverter, 'An object that converts strings from one thing to another.' ),
( 'String Splitter', ClientParsing.StringSplitter, 'An object that breaks strings into smaller strings.' ),
( 'String Sorter', ClientParsing.StringSorter, 'An object that sorts strings.' ),
( 'String Selector/Slicer', ClientParsing.StringSlicer, 'An object that filter-selects from the list of strings. Either absolute index position or a range.' )
( 'String Match', ClientStrings.StringMatch, 'An object that filters strings.' ),
( 'String Converter', ClientStrings.StringConverter, 'An object that converts strings from one thing to another.' ),
( 'String Splitter', ClientStrings.StringSplitter, 'An object that breaks strings into smaller strings.' ),
( 'String Sorter', ClientStrings.StringSorter, 'An object that sorts strings.' ),
( 'String Selector/Slicer', ClientStrings.StringSlicer, 'An object that filter-selects from the list of strings. Either absolute index position or a range.' )
]
try:
@ -1820,15 +1821,15 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
raise HydrusExceptions.VetoException()
if string_processing_step_type == ClientParsing.StringMatch:
if string_processing_step_type == ClientStrings.StringMatch:
example_text = self._single_test_panel.GetStartingText()
string_processing_step = ClientParsing.StringMatch( example_string = example_text )
string_processing_step = ClientStrings.StringMatch( example_string = example_text )
example_text = self._GetExampleTextForStringProcessingStep( string_processing_step )
string_processing_step = ClientParsing.StringMatch( example_string = example_text )
string_processing_step = ClientStrings.StringMatch( example_string = example_text )
else:
@ -1838,33 +1839,33 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
return self._Edit( string_processing_step )
def _Edit( self, string_processing_step: ClientParsing.StringProcessingStep ):
def _Edit( self, string_processing_step: ClientStrings.StringProcessingStep ):
example_text = self._GetExampleTextForStringProcessingStep( string_processing_step )
with ClientGUITopLevelWindowsPanels.DialogEdit( self, 'edit processing step' ) as dlg:
if isinstance( string_processing_step, ClientParsing.StringMatch ):
if isinstance( string_processing_step, ClientStrings.StringMatch ):
test_data = ClientParsing.ParsingTestData( {}, ( example_text, ) )
panel = EditStringMatchPanel( dlg, string_processing_step, test_data = test_data )
elif isinstance( string_processing_step, ClientParsing.StringConverter ):
elif isinstance( string_processing_step, ClientStrings.StringConverter ):
panel = EditStringConverterPanel( dlg, string_processing_step, example_string_override = example_text )
elif isinstance( string_processing_step, ClientParsing.StringSplitter ):
elif isinstance( string_processing_step, ClientStrings.StringSplitter ):
panel = EditStringSplitterPanel( dlg, string_processing_step, example_string = example_text )
elif isinstance( string_processing_step, ClientParsing.StringSorter ):
elif isinstance( string_processing_step, ClientStrings.StringSorter ):
test_data = self._GetExampleTextsForStringSorter( string_processing_step )
panel = EditStringSorterPanel( dlg, string_processing_step, test_data = test_data )
elif isinstance( string_processing_step, ClientParsing.StringSlicer ):
elif isinstance( string_processing_step, ClientStrings.StringSlicer ):
test_data = self._GetExampleTextsForStringSorter( string_processing_step )
@ -1886,12 +1887,12 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
def _ConvertDataToListBoxString( self, string_processing_step: ClientParsing.StringProcessingStep ):
def _ConvertDataToListBoxString( self, string_processing_step: ClientStrings.StringProcessingStep ):
return string_processing_step.ToString( with_type = True )
def _GetExampleTextForStringProcessingStep( self, string_processing_step: ClientParsing.StringProcessingStep ):
def _GetExampleTextForStringProcessingStep( self, string_processing_step: ClientStrings.StringProcessingStep ):
# ultimately rework this to multiline test_data m8, but the panels need it first
@ -1913,7 +1914,7 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
return example_text
def _GetExampleTextsForStringSorter( self, string_processing_step: ClientParsing.StringProcessingStep ):
def _GetExampleTextsForStringSorter( self, string_processing_step: ClientStrings.StringProcessingStep ):
# ultimately rework this to multiline test_data m8
@ -1939,7 +1940,7 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
processing_steps = self._processing_steps.GetData()
string_processor = ClientParsing.StringProcessor()
string_processor = ClientStrings.StringProcessor()
string_processor.SetProcessingSteps( processing_steps )
@ -1961,7 +1962,7 @@ class EditStringProcessorPanel( ClientGUIScrolledPanels.EditPanel ):
return string_processor
def SetValue( self, string_processor: ClientParsing.StringProcessor ):
def SetValue( self, string_processor: ClientStrings.StringProcessor ):
processing_steps = string_processor.GetProcessingSteps()

12
hydrus/client/gui/lists/ClientGUIListCtrl.py
View File

@ -559,6 +559,10 @@ class BetterListCtrl( QW.QTreeWidget ):
self.ProcessDeleteAction()
elif key in ( QC.Qt.Key_Enter, QC.Qt.Key_Return ):
self.ProcessActivateAction()
elif key in ( ord( 'A' ), ord( 'a' ) ) and modifier == QC.Qt.ControlModifier:
self.selectAll()
@ -633,6 +637,14 @@ class BetterListCtrl( QW.QTreeWidget ):
return len( self.selectedItems() ) > 0
def ProcessActivateAction( self ):
if self._activation_callback is not None:
self._activation_callback()
def ProcessDeleteAction( self ):
if self._use_simple_delete:

3
hydrus/client/gui/pages/ClientGUIResults.py
View File

@ -19,9 +19,9 @@ from hydrus.client import ClientApplicationCommand as CAC
from hydrus.client import ClientConstants as CC
from hydrus.client import ClientData
from hydrus.client import ClientFiles
from hydrus.client.media import ClientMedia
from hydrus.client import ClientPaths
from hydrus.client import ClientSearch
from hydrus.client import ClientStrings
from hydrus.client.gui import ClientGUIDragDrop
from hydrus.client.gui import ClientGUICore as CGC
from hydrus.client.gui import ClientGUIDialogs
@ -42,6 +42,7 @@ from hydrus.client.gui import QtPorting as QP
from hydrus.client.gui.canvas import ClientGUICanvas
from hydrus.client.gui.canvas import ClientGUICanvasFrame
from hydrus.client.gui.networking import ClientGUIHydrusNetwork
from hydrus.client.media import ClientMedia
from hydrus.client.metadata import ClientTags
class MediaPanel( ClientMedia.ListeningMediaList, QW.QScrollArea ):

28
hydrus/client/importing/ClientImportFileSeeds.py
View File

@ -385,6 +385,11 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
return None
def GetHashTypesToHashes( self ):
return dict( self._hashes )
def GetPreImportStatusPredictionHash( self, file_import_options: FileImportOptions.FileImportOptions ) -> ClientImportFiles.FileImportStatus:
hash_match_found = False
@ -587,6 +592,19 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
return search_file_seeds
def GetExternalTags( self ):
t = set( self._tags )
t.update( self._external_filterable_tags )
return t
def GetURLs( self ):
return set( self._urls )
def HasHash( self ):
return self.GetHash() is not None
@ -696,6 +714,11 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
return self.status == CC.STATUS_DELETED
def IsLocalFileImport( self ):
return self.file_seed_type == FILE_SEED_TYPE_HDD
def IsProbablyMasterPostURL( self ):
if self.file_seed_type == FILE_SEED_TYPE_URL:
@ -724,6 +747,11 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
return True
def IsURLFileImport( self ):
return self.file_seed_type == FILE_SEED_TYPE_URL
def Normalise( self ):
if self.file_seed_type == FILE_SEED_TYPE_URL:

6
hydrus/client/importing/ClientImportGallery.py
View File

@ -1467,7 +1467,7 @@ class MultipleGalleryImport( HydrusSerialisable.SerialisableBase ):
def PendSubscriptionGapDownloader( self, gug_key_and_name, query_text, file_limit ):
def PendSubscriptionGapDownloader( self, gug_key_and_name, query_text, file_import_options, tag_import_options, file_limit ):
with self._lock:
@ -1493,8 +1493,8 @@ class MultipleGalleryImport( HydrusSerialisable.SerialisableBase ):
gallery_import.SetFileLimit( file_limit )
gallery_import.SetFileImportOptions( self._file_import_options )
gallery_import.SetTagImportOptions( self._tag_import_options )
gallery_import.SetFileImportOptions( file_import_options )
gallery_import.SetTagImportOptions( tag_import_options )
publish_to_page = False

12
hydrus/client/importing/ClientImportSubscriptions.py
View File

@ -1,4 +1,3 @@
import gc
import os
import random
import threading
@ -226,7 +225,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
message += os.linesep
message += 'Either a user uploaded a lot of files to that query in a short period, in which case there is a gap in your subscription you may wish to fill, or the site has just changed its URL format, in which case you may see several of these messages for this site over the coming weeks, and you should ignore them.'
call = HydrusData.Call( HG.client_controller.pub, 'make_new_subscription_gap_downloader', self._gug_key_and_name, query_text, file_limit * 5 )
call = HydrusData.Call( HG.client_controller.pub, 'make_new_subscription_gap_downloader', self._gug_key_and_name, query_text, self._file_import_options.Duplicate(), self._tag_import_options.Duplicate(), file_limit * 5 )
call.SetLabel( 'start a new downloader for this to fill in the gap!' )
@ -1707,6 +1706,8 @@ class SubscriptionsManager( object ):
self._wake_event = threading.Event()
self._big_pauser = HydrusData.BigJobPauser( wait_time = 0.8 )
self._controller.sub( self, 'Shutdown', 'shutdown' )
@ -1726,11 +1727,6 @@ class SubscriptionsManager( object ):
if done_some:
gc.collect()
def _GetMainLoopWaitTime( self ):
@ -1901,6 +1897,8 @@ class SubscriptionsManager( object ):
wait_time = self._GetMainLoopWaitTime()
self._big_pauser.Pause()
self._wake_event.wait( wait_time )
self._wake_event.clear()

24
hydrus/client/networking/ClientLocalServerResources.py
View File

@ -96,7 +96,17 @@ def ParseClientAPIPOSTByteArgs( args ):
try:
v = bytes.fromhex( parsed_request_args[ var_name ] )
raw_value = parsed_request_args[ var_name ]
# In JSON, if someone puts 'null' for an optional value, treat that as 'did not enter anything'
if raw_value is None:
del parsed_request_args[ var_name ]
continue
v = bytes.fromhex( raw_value )
if len( v ) == 0:
@ -120,7 +130,17 @@ def ParseClientAPIPOSTByteArgs( args ):
try:
v_list = [ bytes.fromhex( hash_hex ) for hash_hex in parsed_request_args[ var_name ] ]
raw_value = parsed_request_args[ var_name ]
# In JSON, if someone puts 'null' for an optional value, treat that as 'did not enter anything'
if raw_value is None:
del parsed_request_args[ var_name ]
continue
v_list = [ bytes.fromhex( hash_hex ) for hash_hex in raw_value ]
v_list = [ v for v in v_list if len( v ) > 0 ]

259
hydrus/client/networking/ClientNetworkingDomain.py
View File

@ -15,7 +15,7 @@ from hydrus.core import HydrusSerialisable
from hydrus.core.networking import HydrusNetworking
from hydrus.client import ClientConstants as CC
from hydrus.client import ClientParsing
from hydrus.client import ClientStrings
from hydrus.client import ClientThreading
from hydrus.client.networking import ClientNetworkingContexts
@ -40,12 +40,6 @@ def AddCookieToSession( session, name, value, domain, path, expires, secure = Fa
session.cookies.set_cookie( cookie )
def AlphabetiseQueryText( query_text ):
( query_dict, param_order ) = ConvertQueryTextToDict( query_text )
return ConvertQueryDictToText( query_dict )
def ConvertDomainIntoAllApplicableDomains( domain, discard_www = True ):
# is an ip address or localhost, possibly with a port
@ -119,21 +113,53 @@ def ConvertHTTPToHTTPS( url ):
raise Exception( 'Given a url that did not have a scheme!' )
def ConvertQueryDictToText( query_dict, param_order = None ):
def ConvertQueryDictToText( query_dict, single_value_parameters, param_order = None ):
# we now do everything with requests, which does all the unicode -> %20 business naturally, phew
# we still want to call str explicitly to coerce integers and so on that'll slip in here and there
if param_order is None:
param_pairs = sorted( query_dict.items() )
param_order = sorted( query_dict.keys() )
else:
single_value_parameters = list( single_value_parameters )
single_value_parameters.sort()
param_pairs = [ ( key, query_dict[ key ] ) for key in param_order if key in query_dict ]
for i in range( len( single_value_parameters ) ):
param_order.append( None )
query_text = '&'.join( ( str( key ) + '=' + str( value ) for ( key, value ) in param_pairs ) )
params = []
single_value_parameter_index = 0
for key in param_order:
if key is None:
try:
params.append( single_value_parameters[ single_value_parameter_index ] )
except IndexError:
continue
single_value_parameter_index += 1
else:
if key in query_dict:
params.append( '{}={}'.format( key, query_dict[ key ] ) )
query_text = '&'.join( params )
return query_text
@ -153,6 +179,7 @@ def ConvertQueryTextToDict( query_text ):
param_order = []
query_dict = {}
single_value_parameters = []
pairs = query_text.split( '&' )
@ -162,7 +189,38 @@ def ConvertQueryTextToDict( query_text ):
# for the moment, ignore tracker bugs and so on that have only key and no value
if len( result ) == 2:
if len( result ) == 1:
( value, ) = result
if value == '':
continue
try:
unquoted_value = urllib.parse.unquote( value )
if True not in ( bad_char in unquoted_value for bad_char in bad_chars ):
requoted_value = urllib.parse.quote( unquoted_value )
if requoted_value == value:
value = unquoted_value
except:
pass
single_value_parameters.append( value )
param_order.append( None )
elif len( result ) == 2:
( key, value ) = result
@ -210,7 +268,7 @@ def ConvertQueryTextToDict( query_text ):
return ( query_dict, param_order )
return ( query_dict, single_value_parameters, param_order )
def ConvertURLClassesIntoAPIPairs( url_classes ):
@ -1772,7 +1830,11 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
netloc = p.netloc
path = p.path
params = p.params
query = AlphabetiseQueryText( p.query )
( query_dict, single_value_parameters, param_order ) = ConvertQueryTextToDict( p.query )
query = ConvertQueryDictToText( query_dict, single_value_parameters )
fragment = ''
r = urllib.parse.ParseResult( scheme, netloc, path, params, query, fragment )
@ -2979,7 +3041,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_URL_CLASS
SERIALISABLE_NAME = 'URL Class'
SERIALISABLE_VERSION = 10
SERIALISABLE_VERSION = 11
def __init__(
self,
@ -2990,6 +3052,8 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
netloc = 'hostname.com',
path_components = None,
parameters = None,
has_single_value_parameters = False,
single_value_parameters_string_match = None,
header_overrides = None,
api_lookup_converter = None,
send_referral_url = SEND_REFERRAL_URL_ONLY_IF_PROVIDED,
@ -3014,16 +3078,21 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
path_components = []
path_components.append( ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
path_components.append( ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
if parameters is None:
parameters = {}
parameters[ 's' ] = ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ), None )
parameters[ 'id' ] = ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FLEXIBLE, match_value = ClientParsing.NUMERIC, example_string = '123456' ), None )
parameters[ 's' ] = ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ), None )
parameters[ 'id' ] = ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ), None )
if single_value_parameters_string_match is None:
single_value_parameters_string_match = ClientStrings.StringMatch()
if header_overrides is None:
@ -3033,12 +3102,12 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
if api_lookup_converter is None:
api_lookup_converter = ClientParsing.StringConverter( example_string = 'https://hostname.com/post/page.php?id=123456&s=view' )
api_lookup_converter = ClientStrings.StringConverter( example_string = 'https://hostname.com/post/page.php?id=123456&s=view' )
if referral_url_converter is None:
referral_url_converter = ClientParsing.StringConverter( example_string = 'https://hostname.com/post/page.php?id=123456&s=view' )
referral_url_converter = ClientStrings.StringConverter( example_string = 'https://hostname.com/post/page.php?id=123456&s=view' )
# if the args are not serialisable stuff, lets overwrite here
@ -3062,6 +3131,8 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
self._path_components = path_components
self._parameters = parameters
self._has_single_value_parameters = has_single_value_parameters
self._single_value_parameters_string_match = single_value_parameters_string_match
self._header_overrides = header_overrides
self._api_lookup_converter = api_lookup_converter
@ -3145,7 +3216,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
def _ClipAndFleshOutQuery( self, query, allow_clip = True ):
( query_dict, param_order ) = ConvertQueryTextToDict( query )
( query_dict, single_value_parameters, param_order ) = ConvertQueryTextToDict( query )
if allow_clip:
@ -3164,6 +3235,8 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
query_dict[ key ] = default
param_order.append( key )
@ -3172,7 +3245,12 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
param_order = None
query = ConvertQueryDictToText( query_dict, param_order = param_order )
if not self._has_single_value_parameters:
single_value_parameters = []
query = ConvertQueryDictToText( query_dict, single_value_parameters, param_order = param_order )
return query
@ -3181,25 +3259,63 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
serialisable_url_class_key = self._url_class_key.hex()
serialisable_path_components = [ ( string_match.GetSerialisableTuple(), default ) for ( string_match, default ) in self._path_components ]
serialisable_parameters = [ ( key, ( string_match.GetSerialisableTuple(), default ) ) for ( key, ( string_match, default ) ) in list(self._parameters.items()) ]
serialisable_parameters = [ ( key, ( string_match.GetSerialisableTuple(), default ) ) for ( key, ( string_match, default ) ) in self._parameters.items() ]
serialisable_single_value_parameters_string_match = self._single_value_parameters_string_match.GetSerialisableTuple()
serialisable_header_overrides = list( self._header_overrides.items() )
serialisable_api_lookup_converter = self._api_lookup_converter.GetSerialisableTuple()
serialisable_referral_url_converter = self._referral_url_converter.GetSerialisableTuple()
booleans = ( self._match_subdomains, self._keep_matched_subdomains, self._alphabetise_get_parameters, self._can_produce_multiple_files, self._should_be_associated_with_files, self._keep_fragment )
return ( serialisable_url_class_key, self._url_type, self._preferred_scheme, self._netloc, booleans, serialisable_path_components, serialisable_parameters, serialisable_header_overrides, serialisable_api_lookup_converter, self._send_referral_url, serialisable_referral_url_converter, self._gallery_index_type, self._gallery_index_identifier, self._gallery_index_delta, self._example_url )
return (
serialisable_url_class_key,
self._url_type,
self._preferred_scheme,
self._netloc,
booleans,
serialisable_path_components,
serialisable_parameters,
self._has_single_value_parameters,
serialisable_single_value_parameters_string_match,
serialisable_header_overrides,
serialisable_api_lookup_converter,
self._send_referral_url,
serialisable_referral_url_converter,
self._gallery_index_type,
self._gallery_index_identifier,
self._gallery_index_delta,
self._example_url
)
def _InitialiseFromSerialisableInfo( self, serialisable_info ):
( serialisable_url_class_key, self._url_type, self._preferred_scheme, self._netloc, booleans, serialisable_path_components, serialisable_parameters, serialisable_header_overrides, serialisable_api_lookup_converter, self._send_referral_url, serialisable_referral_url_converter, self._gallery_index_type, self._gallery_index_identifier, self._gallery_index_delta, self._example_url ) = serialisable_info
(
serialisable_url_class_key,
self._url_type,
self._preferred_scheme,
self._netloc,
booleans,
serialisable_path_components,
serialisable_parameters,
self._has_single_value_parameters,
serialisable_single_value_parameters_string_match,
serialisable_header_overrides,
serialisable_api_lookup_converter,
self._send_referral_url,
serialisable_referral_url_converter,
self._gallery_index_type,
self._gallery_index_identifier,
self._gallery_index_delta,
self._example_url
) = serialisable_info
( self._match_subdomains, self._keep_matched_subdomains, self._alphabetise_get_parameters, self._can_produce_multiple_files, self._should_be_associated_with_files, self._keep_fragment ) = booleans
self._url_class_key = bytes.fromhex( serialisable_url_class_key )
self._path_components = [ ( HydrusSerialisable.CreateFromSerialisableTuple( serialisable_string_match ), default ) for ( serialisable_string_match, default ) in serialisable_path_components ]
self._parameters = { key : ( HydrusSerialisable.CreateFromSerialisableTuple( serialisable_string_match ), default ) for ( key, ( serialisable_string_match, default ) ) in serialisable_parameters }
self._single_value_parameters_string_match = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_single_value_parameters_string_match )
self._header_overrides = dict( serialisable_header_overrides )
self._api_lookup_converter = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_api_lookup_converter )
self._referral_url_converter = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_referral_url_converter )
@ -3215,7 +3331,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
serialisable_url_class_key = url_class_key.hex()
api_lookup_converter = ClientParsing.StringConverter( example_string = example_url )
api_lookup_converter = ClientStrings.StringConverter( example_string = example_url )
serialisable_api_lookup_converter = api_lookup_converter.GetSerialisableTuple()
@ -3289,7 +3405,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
( serialisable_url_class_key, url_type, preferred_scheme, netloc, match_subdomains, keep_matched_subdomains, serialisable_path_components, serialisable_parameters, serialisable_api_lookup_converter, can_produce_multiple_files, should_be_associated_with_files, gallery_index_type, gallery_index_identifier, gallery_index_delta, example_url ) = old_serialisable_info
send_referral_url = SEND_REFERRAL_URL_ONLY_IF_PROVIDED
referral_url_converter = ClientParsing.StringConverter( example_string = 'https://hostname.com/post/page.php?id=123456&s=view' )
referral_url_converter = ClientStrings.StringConverter( example_string = 'https://hostname.com/post/page.php?id=123456&s=view' )
serialisable_referrel_url_converter = referral_url_converter.GetSerialisableTuple()
@ -3335,6 +3451,38 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
return ( 10, new_serialisable_info )
if version == 10:
( serialisable_url_class_key, url_type, preferred_scheme, netloc, booleans, serialisable_path_components, serialisable_parameters, serialisable_header_overrides, serialisable_api_lookup_converter, send_referral_url, serialisable_referrel_url_converter, gallery_index_type, gallery_index_identifier, gallery_index_delta, example_url ) = old_serialisable_info
has_single_value_parameters = False
single_value_parameters_string_match = ClientStrings.StringMatch()
serialisable_single_value_parameters_match = single_value_parameters_string_match.GetSerialisableTuple()
new_serialisable_info = (
serialisable_url_class_key,
url_type,
preferred_scheme,
netloc,
booleans,
serialisable_path_components,
serialisable_parameters,
has_single_value_parameters,
serialisable_single_value_parameters_match,
serialisable_header_overrides,
serialisable_api_lookup_converter,
send_referral_url,
serialisable_referrel_url_converter,
gallery_index_type,
gallery_index_identifier,
gallery_index_delta,
example_url
)
return ( 11, new_serialisable_info )
def AlphabetiseGetParameters( self ):
@ -3455,7 +3603,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
page_index_name = self._gallery_index_identifier
( query_dict, param_order ) = ConvertQueryTextToDict( query )
( query_dict, single_value_parameters, param_order ) = ConvertQueryTextToDict( query )
if page_index_name not in query_dict:
@ -3480,7 +3628,12 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
param_order = None
query = ConvertQueryDictToText( query_dict, param_order = param_order )
if not self._has_single_value_parameters:
single_value_parameters = []
query = ConvertQueryDictToText( query_dict, single_value_parameters, param_order = param_order )
else:
@ -3534,6 +3687,11 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
return 'URL Class "' + self._name + '" - ' + ConvertURLIntoDomain( self.GetExampleURL() )
def GetSingleValueParameterData( self ):
return ( self._has_single_value_parameters, self._single_value_parameters_string_match )
def GetSortingComplexityKey( self ):
# we sort url classes so that
@ -3649,9 +3807,9 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
self._url_class_key = HydrusData.GenerateKey()
def SetExampleURL( self, example_url ):
def SetAlphabetiseGetParameters( self, alphabetise_get_parameters: bool ):
self._example_url = example_url
self._alphabetise_get_parameters = alphabetise_get_parameters
def SetClassKey( self, match_key ):
@ -3659,6 +3817,17 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
self._url_class_key = match_key
def SetExampleURL( self, example_url ):
self._example_url = example_url
def SetSingleValueParameterData( self, has_single_value_parameters: bool, single_value_parameters_string_match: ClientStrings.StringMatch ):
self._has_single_value_parameters = has_single_value_parameters
self._single_value_parameters_string_match = single_value_parameters_string_match
def SetURLBooleans(
self,
match_subdomains: bool,
@ -3731,9 +3900,9 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
( url_parameters, param_order ) = ConvertQueryTextToDict( p.query )
( url_parameters, single_value_parameters, param_order ) = ConvertQueryTextToDict( p.query )
for ( key, ( string_match, default ) ) in list(self._parameters.items()):
for ( key, ( string_match, default ) ) in self._parameters.items():
if key not in url_parameters:
@ -3759,6 +3928,26 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
if self._has_single_value_parameters:
if len( single_value_parameters ) == 0:
raise HydrusExceptions.URLClassException( 'Was expecting single-value parameter(s), but this URL did not seem to have any.' )
for single_value_parameter in single_value_parameters:
try:
self._single_value_parameters_string_match.Test( single_value_parameter )
except HydrusExceptions.StringMatchException as e:
raise HydrusExceptions.URLClassException( str( e ) )
def ToTuple( self ):

13
hydrus/client/networking/ClientNetworkingLogin.py
View File

@ -12,6 +12,7 @@ from hydrus.core import HydrusSerialisable
from hydrus.client import ClientConstants as CC
from hydrus.client import ClientParsing
from hydrus.client import ClientStrings
from hydrus.client import ClientThreading
from hydrus.client.networking import ClientNetworkingContexts
from hydrus.client.networking import ClientNetworkingDomain
@ -818,7 +819,7 @@ class LoginCredentialDefinition( HydrusSerialisable.SerialisableBaseNamed ):
if string_match is None:
string_match = ClientParsing.StringMatch()
string_match = ClientStrings.StringMatch()
HydrusSerialisable.SerialisableBaseNamed.__init__( self, name )
@ -1100,7 +1101,7 @@ class LoginScriptDomain( HydrusSerialisable.SerialisableBaseNamed ):
for ( name, value_string_match ) in list(old_required_cookies_info.items()):
key_string_match = ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = name, example_string = name )
key_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = name, example_string = name )
new_required_cookies_info[ key_string_match ] = value_string_match
@ -1533,7 +1534,7 @@ class LoginStep( HydrusSerialisable.SerialisableBaseNamed ):
for ( name, value_string_match ) in list(old_required_cookies_info.items()):
key_string_match = ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = name, example_string = name )
key_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = name, example_string = name )
new_required_cookies_info[ key_string_match ] = value_string_match
@ -1649,9 +1650,11 @@ class LoginStep( HydrusSerialisable.SerialisableBaseNamed ):
params = ''
fragment = ''
single_value_parameters = []
if self._method == 'GET':
query = ClientNetworkingDomain.ConvertQueryDictToText( query_dict )
query = ClientNetworkingDomain.ConvertQueryDictToText( query_dict, single_value_parameters )
body = None
test_result_body = ''
@ -1659,7 +1662,7 @@ class LoginStep( HydrusSerialisable.SerialisableBaseNamed ):
query = ''
body = query_dict
test_result_body = ClientNetworkingDomain.ConvertQueryDictToText( query_dict )
test_result_body = ClientNetworkingDomain.ConvertQueryDictToText( query_dict, single_value_parameters )
r = urllib.parse.ParseResult( scheme, netloc, path, params, query, fragment )

2
hydrus/core/HydrusConstants.py
View File

@ -81,7 +81,7 @@ options = {}
# Misc
NETWORK_VERSION = 20
SOFTWARE_VERSION = 455
SOFTWARE_VERSION = 456
CLIENT_API_VERSION = 20
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )

3
hydrus/core/HydrusController.py
View File

@ -1,5 +1,4 @@
import collections
import gc
import os
import random
import sys
@ -619,8 +618,6 @@ class HydrusController( object ):
def MaintainMemorySlow( self ):
gc.collect()
HydrusPaths.CleanUpOldTempPaths()
self._MaintainCallToThreads()

20
hydrus/core/HydrusDBBase.py
View File

@ -416,9 +416,23 @@ class DBCursorTransactionWrapper( DBBase ):
def Save( self ):
self._Execute( 'RELEASE hydrus_savepoint;' )
self._Execute( 'SAVEPOINT hydrus_savepoint;' )
if self._in_transaction:
try:
self._Execute( 'RELEASE hydrus_savepoint;' )
except sqlite3.OperationalError:
HydrusData.Print( 'Tried to release a database savepoint, but failed!' )
self._Execute( 'SAVEPOINT hydrus_savepoint;' )
else:
HydrusData.Print( 'Received a call to save, but was not in a transaction!' )
def TimeToCommit( self ):

1
hydrus/core/HydrusGlobals.py
View File

@ -64,6 +64,7 @@ phash_generation_report_mode = False
network_report_mode = False
pubsub_report_mode = False
daemon_report_mode = False
mpv_report_mode = False
force_idle_mode = False
no_page_limit_mode = False
thumbnail_debug_mode = False

3
hydrus/core/HydrusPaths.py
View File

@ -1,4 +1,3 @@
import gc
import os
import psutil
import re
@ -99,8 +98,6 @@ def CleanUpTempPath( os_file_handle, temp_path ):
except OSError:
gc.collect()
try:
os.close( os_file_handle )

2
hydrus/core/networking/HydrusNetwork.py
View File

@ -2568,6 +2568,8 @@ class ServerServiceRestricted( ServerService ):
dictionary[ 'bandwidth_rules' ] = self._bandwidth_rules
dictionary[ 'service_options' ] = self._service_options
dictionary[ 'server_message' ] = self._server_message
return dictionary

3
hydrus/core/networking/HydrusNetworkVariableHandling.py
View File

@ -398,7 +398,8 @@ class ParsedRequestArguments( dict ):
def GetValue( self, key, expected_type, expected_list_type = None, default_value = None ):
if key in self:
# not None because in JSON sometimes people put 'null' to mean 'did not enter this optional parameter'
if key in self and self[ key ] is not None:
value = self[ key ]

96
hydrus/test/TestClientNetworking.py
View File

@ -11,7 +11,7 @@ from hydrus.core import HydrusGlobals as HG
from hydrus.core.networking import HydrusNetworking
from hydrus.client import ClientConstants as CC
from hydrus.client import ClientParsing
from hydrus.client import ClientStrings
from hydrus.client import ClientServices
from hydrus.client.networking import ClientNetworking
from hydrus.client.networking import ClientNetworkingBandwidth
@ -240,13 +240,13 @@ class TestNetworkingDomain( unittest.TestCase ):
path_components = []
path_components.append( ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
path_components.append( ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
parameters = {}
parameters[ 's' ] = ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ), None )
parameters[ 'id' ] = ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FLEXIBLE, match_value = ClientParsing.NUMERIC, example_string = '123456' ), None )
parameters[ 's' ] = ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ), None )
parameters[ 'id' ] = ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ), None )
send_referral_url = ClientNetworkingDomain.SEND_REFERRAL_URL_ONLY_IF_PROVIDED
referral_url_converter = None
@ -305,9 +305,9 @@ class TestNetworkingDomain( unittest.TestCase ):
conversions = []
conversions.append( ( ClientParsing.STRING_CONVERSION_REGEX_SUB, ( 'testbooru.cx', 'replace.com' ) ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_REGEX_SUB, ( 'testbooru.cx', 'replace.com' ) ) )
referral_url_converter = ClientParsing.StringConverter( conversions, good_url )
referral_url_converter = ClientStrings.StringConverter( conversions, good_url )
send_referral_url = ClientNetworkingDomain.SEND_REFERRAL_URL_CONVERTER_IF_NONE_PROVIDED
@ -344,8 +344,8 @@ class TestNetworkingDomain( unittest.TestCase ):
path_components = []
path_components.append( ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = 'file', example_string = 'file' ), None ) )
path_components.append( ( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_ANY ), None ) )
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'file', example_string = 'file' ), None ) )
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_ANY ), None ) )
parameters = {}
@ -372,6 +372,84 @@ class TestNetworkingDomain( unittest.TestCase ):
self.assertEqual( url_class.Normalise( example_url ), example_url )
# single-value params test
single_value_good_url = 'https://testbooru.cx/post/page.php?id=123456&token&s=view'
single_value_bad_url = 'https://testbooru.cx/post/page.php?id=123456&bad_token&s=view'
single_value_missing_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
single_value_good_url_multiple = 'https://testbooru.cx/post/page.php?id=123456&token1&token2&s=view&token0'
single_value_good_url_alphabetical_normalised = 'https://testbooru.cx/post/page.php?id=123456&s=view&token'
single_value_good_url_multiple_alphabetical_normalised = 'https://testbooru.cx/post/page.php?id=123456&s=view&token0&token1&token2'
name = 'single value lad'
url_type = HC.URL_TYPE_POST
preferred_scheme = 'https'
netloc = 'testbooru.cx'
alphabetise_get_parameters = True
match_subdomains = False
keep_matched_subdomains = False
can_produce_multiple_files = False
should_be_associated_with_files = True
keep_fragment = False
path_components = []
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
parameters = {}
parameters[ 's' ] = ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ), None )
parameters[ 'id' ] = ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ), None )
has_single_value_parameters = True
single_value_parameters_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_REGEX, match_value = '^token.*', example_string = 'token1' )
example_url = single_value_good_url
url_class = ClientNetworkingDomain.URLClass(
name,
url_type = url_type,
preferred_scheme = preferred_scheme,
netloc = netloc,
path_components = path_components,
parameters = parameters,
has_single_value_parameters = has_single_value_parameters,
single_value_parameters_string_match = single_value_parameters_string_match,
send_referral_url = send_referral_url,
referral_url_converter = referral_url_converter,
gallery_index_type = gallery_index_type,
gallery_index_identifier = gallery_index_identifier,
gallery_index_delta = gallery_index_delta,
example_url = example_url
)
url_class.SetURLBooleans( match_subdomains, keep_matched_subdomains, alphabetise_get_parameters, can_produce_multiple_files, should_be_associated_with_files, keep_fragment )
self.assertEqual( url_class.Normalise( single_value_good_url ), single_value_good_url_alphabetical_normalised )
self.assertEqual( url_class.Normalise( single_value_good_url_multiple ), single_value_good_url_multiple_alphabetical_normalised )
self.assertEqual( url_class.Matches( single_value_good_url ), True )
self.assertEqual( url_class.Matches( single_value_good_url_alphabetical_normalised ), True )
self.assertEqual( url_class.Matches( single_value_good_url_multiple ), True )
self.assertEqual( url_class.Matches( single_value_good_url_multiple_alphabetical_normalised ), True )
self.assertEqual( url_class.Matches( single_value_bad_url ), False )
self.assertEqual( url_class.Matches( single_value_missing_url ), False )
url_class.SetAlphabetiseGetParameters( False )
self.assertEqual( url_class.Normalise( single_value_good_url ), single_value_good_url )
self.assertEqual( url_class.Normalise( single_value_good_url_multiple ), single_value_good_url_multiple )
self.assertEqual( url_class.Matches( single_value_good_url ), True )
self.assertEqual( url_class.Matches( single_value_good_url_alphabetical_normalised ), True )
self.assertEqual( url_class.Matches( single_value_good_url_multiple ), True )
self.assertEqual( url_class.Matches( single_value_good_url_multiple_alphabetical_normalised ), True )
self.assertEqual( url_class.Matches( single_value_bad_url ), False )
self.assertEqual( url_class.Matches( single_value_missing_url ), False )
class TestNetworkingEngine( unittest.TestCase ):

180
hydrus/test/TestClientParsing.py
View File

@ -5,121 +5,121 @@ import unittest
from hydrus.core import HydrusConstants as HC
from hydrus.core import HydrusExceptions
from hydrus.client import ClientParsing
from hydrus.client import ClientStrings
class TestStringConverter( unittest.TestCase ):
def test_basics( self ):
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, 1 ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, 1 ) ] )
self.assertEqual( string_converter.Convert( '0123456789' ), '123456789' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_END, 1 ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_END, 1 ) ] )
self.assertEqual( string_converter.Convert( '0123456789' ), '012345678' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, 7 ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, 7 ) ] )
self.assertEqual( string_converter.Convert( '0123456789' ), '0123456' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_END, 7 ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_END, 7 ) ] )
self.assertEqual( string_converter.Convert( '0123456789' ), '3456789' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_PREPEND_TEXT, 'abc' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_PREPEND_TEXT, 'abc' ) ] )
self.assertEqual( string_converter.Convert( '0123456789' ), 'abc0123456789' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_APPEND_TEXT, 'xyz' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_APPEND_TEXT, 'xyz' ) ] )
self.assertEqual( string_converter.Convert( '0123456789' ), '0123456789xyz' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_ENCODE, 'url percent encoding' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_ENCODE, 'url percent encoding' ) ] )
self.assertEqual( string_converter.Convert( '01234 56789' ), '01234%2056789' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_DECODE, 'url percent encoding' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_DECODE, 'url percent encoding' ) ] )
self.assertEqual( string_converter.Convert( '01234%2056789' ), '01234 56789' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_ENCODE, 'unicode escape characters' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_ENCODE, 'unicode escape characters' ) ] )
self.assertEqual( string_converter.Convert( '01234\u039456789' ), '01234\\u039456789' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_DECODE, 'unicode escape characters' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_DECODE, 'unicode escape characters' ) ] )
self.assertEqual( string_converter.Convert( '01234\\u039456789' ), '01234\u039456789' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_ENCODE, 'html entities' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_ENCODE, 'html entities' ) ] )
self.assertEqual( string_converter.Convert( '01234&56789' ), '01234&amp;56789' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_DECODE, 'html entities' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_DECODE, 'html entities' ) ] )
self.assertEqual( string_converter.Convert( '01234&amp;56789' ), '01234&56789' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_ENCODE, 'hex' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_ENCODE, 'hex' ) ] )
self.assertEqual( string_converter.Convert( b'\xe5\xafW\xa6\x87\xf0\x89\x89O^\xce\xdeP\x04\x94X' ), 'e5af57a687f089894f5ecede50049458' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_ENCODE, 'base64' ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_ENCODE, 'base64' ) ] )
self.assertEqual( string_converter.Convert( b'\xe5\xafW\xa6\x87\xf0\x89\x89O^\xce\xdeP\x04\x94X' ), '5a9XpofwiYlPXs7eUASUWA==' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_REVERSE, None ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_REVERSE, None ) ] )
self.assertEqual( string_converter.Convert( '0123456789' ), '9876543210' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_REGEX_SUB, ( '\\d', 'd' ) ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_REGEX_SUB, ( '\\d', 'd' ) ) ] )
self.assertEqual( string_converter.Convert( 'abc123' ), 'abcddd' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_DATE_DECODE, ( '%Y-%m-%d %H:%M:%S', HC.TIMEZONE_GMT, 0 ) ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_DATE_DECODE, ( '%Y-%m-%d %H:%M:%S', HC.TIMEZONE_GMT, 0 ) ) ] )
self.assertEqual( string_converter.Convert( '1970-01-02 00:00:00' ), '86400' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_DATE_ENCODE, ( '%Y-%m-%d %H:%M:%S', 0 ) ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_DATE_ENCODE, ( '%Y-%m-%d %H:%M:%S', 0 ) ) ] )
self.assertEqual( string_converter.Convert( '86400' ), '1970-01-02 00:00:00' )
#
string_converter = ClientParsing.StringConverter( conversions = [ ( ClientParsing.STRING_CONVERSION_INTEGER_ADDITION, 5 ) ] )
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_INTEGER_ADDITION, 5 ) ] )
self.assertEqual( string_converter.Convert( '4' ), '9' )
@ -128,81 +128,81 @@ class TestStringConverter( unittest.TestCase ):
conversions = []
conversions.append( ( ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, 1 ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING, 1 ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), '123456789' )
#
conversions.append( ( ClientParsing.STRING_CONVERSION_REMOVE_TEXT_FROM_END, 1 ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_REMOVE_TEXT_FROM_END, 1 ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), '12345678' )
#
conversions.append( ( ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, 7 ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING, 7 ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), '1234567' )
#
conversions.append( ( ClientParsing.STRING_CONVERSION_CLIP_TEXT_FROM_END, 6 ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_END, 6 ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), '234567' )
#
conversions.append( ( ClientParsing.STRING_CONVERSION_PREPEND_TEXT, 'abc' ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_PREPEND_TEXT, 'abc' ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), 'abc234567' )
#
conversions.append( ( ClientParsing.STRING_CONVERSION_APPEND_TEXT, 'x z' ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_APPEND_TEXT, 'x z' ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), 'abc234567x z' )
#
conversions.append( ( ClientParsing.STRING_CONVERSION_ENCODE, 'url percent encoding' ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_ENCODE, 'url percent encoding' ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), 'abc234567x%20z' )
#
conversions.append( ( ClientParsing.STRING_CONVERSION_DECODE, 'url percent encoding' ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_DECODE, 'url percent encoding' ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), 'abc234567x z' )
#
conversions.append( ( ClientParsing.STRING_CONVERSION_REVERSE, None ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_REVERSE, None ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), 'z x765432cba' )
#
conversions.append( ( ClientParsing.STRING_CONVERSION_REGEX_SUB, ( '\\d', 'd' ) ) )
conversions.append( ( ClientStrings.STRING_CONVERSION_REGEX_SUB, ( '\\d', 'd' ) ) )
string_converter = ClientParsing.StringConverter( conversions = conversions )
string_converter = ClientStrings.StringConverter( conversions = conversions )
self.assertEqual( string_converter.Convert( '0123456789' ), 'z xddddddcba' )
@ -211,7 +211,7 @@ class TestStringMatch( unittest.TestCase ):
def test_basics( self ):
all_string_match = ClientParsing.StringMatch()
all_string_match = ClientStrings.StringMatch()
self.assertTrue( all_string_match.Matches( '123' ) )
self.assertTrue( all_string_match.Matches( 'abc' ) )
@ -219,7 +219,7 @@ class TestStringMatch( unittest.TestCase ):
#
min_string_match = ClientParsing.StringMatch( min_chars = 4 )
min_string_match = ClientStrings.StringMatch( min_chars = 4 )
self.assertFalse( min_string_match.Matches( '123' ) )
self.assertFalse( min_string_match.Matches( 'abc' ) )
@ -227,7 +227,7 @@ class TestStringMatch( unittest.TestCase ):
#
max_string_match = ClientParsing.StringMatch( max_chars = 4 )
max_string_match = ClientStrings.StringMatch( max_chars = 4 )
self.assertTrue( max_string_match.Matches( '123' ) )
self.assertTrue( max_string_match.Matches( 'abc' ) )
@ -235,7 +235,7 @@ class TestStringMatch( unittest.TestCase ):
#
min_max_string_match = ClientParsing.StringMatch( min_chars = 4, max_chars = 10 )
min_max_string_match = ClientStrings.StringMatch( min_chars = 4, max_chars = 10 )
self.assertFalse( min_max_string_match.Matches( '123' ) )
self.assertFalse( min_max_string_match.Matches( 'abc' ) )
@ -243,7 +243,7 @@ class TestStringMatch( unittest.TestCase ):
#
alpha_string_match = ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FLEXIBLE, match_value = ClientParsing.ALPHA )
alpha_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.ALPHA )
self.assertFalse( alpha_string_match.Matches( '123' ) )
self.assertTrue( alpha_string_match.Matches( 'abc' ) )
@ -251,7 +251,7 @@ class TestStringMatch( unittest.TestCase ):
#
alphanum_string_match = ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FLEXIBLE, match_value = ClientParsing.ALPHANUMERIC )
alphanum_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.ALPHANUMERIC )
self.assertTrue( alphanum_string_match.Matches( '123' ) )
self.assertTrue( alphanum_string_match.Matches( 'abc' ) )
@ -259,7 +259,7 @@ class TestStringMatch( unittest.TestCase ):
#
num_string_match = ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FLEXIBLE, match_value = ClientParsing.NUMERIC )
num_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC )
self.assertTrue( num_string_match.Matches( '123' ) )
self.assertFalse( num_string_match.Matches( 'abc' ) )
@ -267,7 +267,7 @@ class TestStringMatch( unittest.TestCase ):
#
fixed_string_match = ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FIXED, match_value = '123' )
fixed_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = '123' )
self.assertTrue( fixed_string_match.Matches( '123' ) )
self.assertFalse( fixed_string_match.Matches( 'abc' ) )
@ -275,7 +275,7 @@ class TestStringMatch( unittest.TestCase ):
#
re_string_match = ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_REGEX, match_value = '\\d' )
re_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_REGEX, match_value = '\\d' )
self.assertTrue( re_string_match.Matches( '123' ) )
self.assertFalse( re_string_match.Matches( 'abc' ) )
@ -301,117 +301,117 @@ class TestStringSlicer( unittest.TestCase ):
#
slicer = ClientParsing.StringSlicer( index_start = 0, index_end = 1 )
slicer = ClientStrings.StringSlicer( index_start = 0, index_end = 1 )
self.assertEqual( slicer.Slice( test_list ), [ a ] )
self.assertEqual( slicer.ToString(), 'selecting the 1st string' )
slicer = ClientParsing.StringSlicer( index_start = 3, index_end = 4 )
slicer = ClientStrings.StringSlicer( index_start = 3, index_end = 4 )
self.assertEqual( slicer.Slice( test_list ), [ d ] )
self.assertEqual( slicer.ToString(), 'selecting the 4th string' )
slicer = ClientParsing.StringSlicer( index_start = -3, index_end = -2 )
slicer = ClientStrings.StringSlicer( index_start = -3, index_end = -2 )
self.assertEqual( slicer.Slice( test_list ), [ h ] )
self.assertEqual( slicer.ToString(), 'selecting the 3rd from last string' )
slicer = ClientParsing.StringSlicer( index_start = -1 )
slicer = ClientStrings.StringSlicer( index_start = -1 )
self.assertEqual( slicer.Slice( test_list ), [ j ] )
self.assertEqual( slicer.ToString(), 'selecting the last string' )
slicer = ClientParsing.StringSlicer( index_start = 15, index_end = 16 )
slicer = ClientStrings.StringSlicer( index_start = 15, index_end = 16 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting the 16th string' )
slicer = ClientParsing.StringSlicer( index_start = -15, index_end = -14 )
slicer = ClientStrings.StringSlicer( index_start = -15, index_end = -14 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting the 15th from last string' )
#
slicer = ClientParsing.StringSlicer( index_start = 0 )
slicer = ClientStrings.StringSlicer( index_start = 0 )
self.assertEqual( slicer.Slice( test_list ), test_list )
self.assertEqual( slicer.ToString(), 'selecting the 1st string and onwards' )
slicer = ClientParsing.StringSlicer( index_start = 3 )
slicer = ClientStrings.StringSlicer( index_start = 3 )
self.assertEqual( slicer.Slice( test_list ), [ d, e, f, g, h, i, j ] )
self.assertEqual( slicer.ToString(), 'selecting the 4th string and onwards' )
slicer = ClientParsing.StringSlicer( index_start = -3 )
slicer = ClientStrings.StringSlicer( index_start = -3 )
self.assertEqual( slicer.Slice( test_list ), [ h, i, j ] )
self.assertEqual( slicer.ToString(), 'selecting the 3rd from last string and onwards' )
slicer = ClientParsing.StringSlicer( index_start = 15 )
slicer = ClientStrings.StringSlicer( index_start = 15 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting the 16th string and onwards' )
slicer = ClientParsing.StringSlicer( index_start = -15 )
slicer = ClientStrings.StringSlicer( index_start = -15 )
self.assertEqual( slicer.Slice( test_list ), test_list )
self.assertEqual( slicer.ToString(), 'selecting the 15th from last string and onwards' )
#
slicer = ClientParsing.StringSlicer( index_end = 0 )
slicer = ClientStrings.StringSlicer( index_end = 0 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting nothing' )
slicer = ClientParsing.StringSlicer( index_end = 3 )
slicer = ClientStrings.StringSlicer( index_end = 3 )
self.assertEqual( slicer.Slice( test_list ), [ a, b, c ] )
self.assertEqual( slicer.ToString(), 'selecting up to and including the 3rd string' )
slicer = ClientParsing.StringSlicer( index_end = -3 )
slicer = ClientStrings.StringSlicer( index_end = -3 )
self.assertEqual( slicer.Slice( test_list ), [ a, b, c, d, e, f, g ] )
self.assertEqual( slicer.ToString(), 'selecting up to and including the 4th from last string' )
slicer = ClientParsing.StringSlicer( index_end = 15 )
slicer = ClientStrings.StringSlicer( index_end = 15 )
self.assertEqual( slicer.Slice( test_list ), test_list )
self.assertEqual( slicer.ToString(), 'selecting up to and including the 15th string' )
slicer = ClientParsing.StringSlicer( index_end = -15 )
slicer = ClientStrings.StringSlicer( index_end = -15 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting up to and including the 16th from last string' )
#
slicer = ClientParsing.StringSlicer( index_start = 0, index_end = 5 )
slicer = ClientStrings.StringSlicer( index_start = 0, index_end = 5 )
self.assertEqual( slicer.Slice( test_list ), [ a, b, c, d, e ] )
self.assertEqual( slicer.ToString(), 'selecting the 1st string up to and including the 5th string' )
slicer = ClientParsing.StringSlicer( index_start = 3, index_end = 5 )
slicer = ClientStrings.StringSlicer( index_start = 3, index_end = 5 )
self.assertEqual( slicer.Slice( test_list ), [ d, e ] )
self.assertEqual( slicer.ToString(), 'selecting the 4th string up to and including the 5th string' )
slicer = ClientParsing.StringSlicer( index_start = -5, index_end = -3 )
slicer = ClientStrings.StringSlicer( index_start = -5, index_end = -3 )
self.assertEqual( slicer.Slice( test_list ), [ f, g ] )
self.assertEqual( slicer.ToString(), 'selecting the 5th from last string up to and including the 4th from last string' )
slicer = ClientParsing.StringSlicer( index_start = 3, index_end = -3 )
slicer = ClientStrings.StringSlicer( index_start = 3, index_end = -3 )
self.assertEqual( slicer.Slice( test_list ), [ d, e, f, g ] )
self.assertEqual( slicer.ToString(), 'selecting the 4th string up to and including the 4th from last string' )
#
slicer = ClientParsing.StringSlicer( index_start = 3, index_end = 3 )
slicer = ClientStrings.StringSlicer( index_start = 3, index_end = 3 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting nothing' )
slicer = ClientParsing.StringSlicer( index_start = 5, index_end = 3 )
slicer = ClientStrings.StringSlicer( index_start = 5, index_end = 3 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting nothing' )
slicer = ClientParsing.StringSlicer( index_start = -3, index_end = -3 )
slicer = ClientStrings.StringSlicer( index_start = -3, index_end = -3 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting nothing' )
slicer = ClientParsing.StringSlicer( index_start = -3, index_end = -5 )
slicer = ClientStrings.StringSlicer( index_start = -3, index_end = -5 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting nothing' )
#
slicer = ClientParsing.StringSlicer( index_start = 15, index_end = 20 )
slicer = ClientStrings.StringSlicer( index_start = 15, index_end = 20 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting the 16th string up to and including the 20th string' )
slicer = ClientParsing.StringSlicer( index_start = -15, index_end = -12 )
slicer = ClientStrings.StringSlicer( index_start = -15, index_end = -12 )
self.assertEqual( slicer.Slice( test_list ), [] )
self.assertEqual( slicer.ToString(), 'selecting the 15th from last string up to and including the 13th from last string' )
@ -438,48 +438,48 @@ class TestStringSorter( unittest.TestCase ):
sorter = ClientParsing.StringSorter( sort_type = ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC, asc = True, regex = None )
sorter = ClientStrings.StringSorter( sort_type = ClientStrings.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC, asc = True, regex = None )
correct = [ a, b, c, d, e ]
do_sort_test( sorter, correct )
sorter = ClientParsing.StringSorter( sort_type = ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC, asc = False, regex = None )
sorter = ClientStrings.StringSorter( sort_type = ClientStrings.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC, asc = False, regex = None )
correct = [ e, d, c, b, a ]
do_sort_test( sorter, correct )
#
sorter = ClientParsing.StringSorter( sort_type = ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT, asc = True, regex = None )
sorter = ClientStrings.StringSorter( sort_type = ClientStrings.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT, asc = True, regex = None )
correct = [ a, b, c, d, e ]
do_sort_test( sorter, correct )
sorter = ClientParsing.StringSorter( sort_type = ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT, asc = False, regex = None )
sorter = ClientStrings.StringSorter( sort_type = ClientStrings.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT, asc = False, regex = None )
correct = [ e, d, c, b, a ]
do_sort_test( sorter, correct )
#
sorter = ClientParsing.StringSorter( sort_type = ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC, asc = True, regex = '\\d+' )
sorter = ClientStrings.StringSorter( sort_type = ClientStrings.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC, asc = True, regex = '\\d+' )
correct = [ c, b, a, d, e ]
do_sort_test( sorter, correct )
sorter = ClientParsing.StringSorter( sort_type = ClientParsing.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC, asc = False, regex = '\\d+' )
sorter = ClientStrings.StringSorter( sort_type = ClientStrings.CONTENT_PARSER_SORT_TYPE_LEXICOGRAPHIC, asc = False, regex = '\\d+' )
correct = [ d, a, b, c, e ]
do_sort_test( sorter, correct )
#
sorter = ClientParsing.StringSorter( sort_type = ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT, asc = True, regex = '\\d+' )
sorter = ClientStrings.StringSorter( sort_type = ClientStrings.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT, asc = True, regex = '\\d+' )
correct = [ b, a, d, c, e ]
do_sort_test( sorter, correct )
sorter = ClientParsing.StringSorter( sort_type = ClientParsing.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT, asc = False, regex = '\\d+' )
sorter = ClientStrings.StringSorter( sort_type = ClientStrings.CONTENT_PARSER_SORT_TYPE_HUMAN_SORT, asc = False, regex = '\\d+' )
correct = [ c, d, a, b, e ]
do_sort_test( sorter, correct )
@ -489,13 +489,13 @@ class TestStringSplitter( unittest.TestCase ):
def test_basics( self ):
splitter = ClientParsing.StringSplitter( separator = ', ' )
splitter = ClientStrings.StringSplitter( separator = ', ' )
self.assertEqual( splitter.Split( '123' ), [ '123' ] )
self.assertEqual( splitter.Split( '1,2,3' ), [ '1,2,3' ] )
self.assertEqual( splitter.Split( '1, 2, 3' ), [ '1', '2', '3' ] )
splitter = ClientParsing.StringSplitter( separator = ', ', max_splits = 2 )
splitter = ClientStrings.StringSplitter( separator = ', ', max_splits = 2 )
self.assertEqual( splitter.Split( '123' ), [ '123' ] )
self.assertEqual( splitter.Split( '1,2,3' ), [ '1,2,3' ] )
@ -506,7 +506,7 @@ class TestStringProcessor( unittest.TestCase ):
def test_basics( self ):
processor = ClientParsing.StringProcessor()
processor = ClientStrings.StringProcessor()
self.assertEqual( processor.ProcessStrings( [] ), [] )
self.assertEqual( processor.ProcessStrings( [ 'test' ] ), [ 'test' ] )
@ -514,13 +514,13 @@ class TestStringProcessor( unittest.TestCase ):
processing_steps = []
processing_steps.append( ClientParsing.StringSplitter( separator = ',', max_splits = 2 ) )
processing_steps.append( ClientStrings.StringSplitter( separator = ',', max_splits = 2 ) )
processing_steps.append( ClientParsing.StringMatch( match_type = ClientParsing.STRING_MATCH_FLEXIBLE, match_value = ClientParsing.NUMERIC ) )
processing_steps.append( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC ) )
conversions = [ ( ClientParsing.STRING_CONVERSION_APPEND_TEXT, 'abc' ) ]
conversions = [ ( ClientStrings.STRING_CONVERSION_APPEND_TEXT, 'abc' ) ]
processing_steps.append( ClientParsing.StringConverter( conversions = conversions ) )
processing_steps.append( ClientStrings.StringConverter( conversions = conversions ) )
processor.SetProcessingSteps( processing_steps )