Version 569
This commit is contained in:
Hydrus Network Developer
parent
a99ab83ecb
commit
3aa8702972
|
|
@ -7,6 +7,46 @@ title: Changelog
|
|||
!!! note
|
||||
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
|
||||
|
||||
## [Version 569](https://github.com/hydrusnetwork/hydrus/releases/tag/v569)
|
||||
|
||||
### user contributions
|
||||
|
||||
* thanks to a user, fixed a problem with the recent URL changes that caused downloaders examining multi-file posts to only grab the first file
|
||||
* thanks to a user, all the menubar commands that launch a modal dialog are now suffix'd by an ellipsis
|
||||
* thanks to a user, fixed an issue regarding KDE 6 quitting the program as soon as the pre-boot 'your database is missing a location, let's find it' repair dialog was ok'd
|
||||
* thanks to a user, the application icon is fixed in KDE Plasma Wayland (and anything else that pulls icon from .desktop file). if you have been using a hydrus.desktop file and don't see a program icon, you should rename it to `/usr/share/applications/io.github.hydrusnetwork.hydrus.desktop` . more importantly, if you manage a package for hydrus--please output to this file path instead of `hydrus.desktop` if you make one
|
||||
* thanks to a user, updated the `hydrus_client.sh` file to include `"$@"`, which passes parameters given to the .sh file to the .py call
|
||||
|
||||
### more on last week's URL work
|
||||
|
||||
* fixed the 'show the Request URL under "additional urls" submenu' thing on the file log list menu. I screwed up the logic and was effectively testing for when `1 != 1`
|
||||
* the converter that generates a Referral URL now operates on the API/redirect conversion principle too--it normalises the Source URL to its 'Request URL' state--keeping defined ephemeral params and filling in defaults but dropping any extra gubbins not asked for--before applying the conversion
|
||||
* fixed the 'manage url class' dialog to correctly display an example API/redirect-converted URL based on the new _request url_, not the _normalised url_ (so the api/redirect example will now show the new ephemeral params properly). this was working in requests correctly behind the scenes, it was just the example text box in the dialog that was showing wrong
|
||||
* improved the 'is this query text pre-encoded?' test to check for `%hh`, where `h` is a hexadecimal character, instead of the hackier 'is % in it while not followed by whitespace or end of string?'
|
||||
* improved/simplified/optimised the overall procedure that figures out if an entered URL is pre-encoded or not. this routine now only runs at the stage where a URL is ingested and it obeys the `%hh` rule. these ingestion points are currently: the text boxes in a urls downloader/simple downloader page; the 'import new sources' function of file log menus; a URL `ContentParser` in the parsing system; the test box in `manage url classes`; and the main gui's 'import url' landing pad, which is used by the drag and drop system, the clipboard watcher, and the client api's 'import url' command. note that this does not occur on 'manage known urls' editing, where you can do what you want with whatever, and I won't coerce it to anything
|
||||
|
||||
### misc
|
||||
|
||||
* fixed a variety of logical cases around >0, =0, !=0, <0 for the `NumberTest` objects I recently applied to system:duration and elsewhere. when it comes to file searching, files that have 'None' duration are now considered equivalent to files that have an explicit 0 duration in all cases. previously, I was trying to thread a needle where '=0' would find null results but <x would not, and it was a mess. now it all works the same way. if you want to search for 'duration < x' and want to exclude still images, either add a filetype pred or slap on 'has duration'
|
||||
* improved the stability of the manual file exporter process. it was consulting an object in a thread that it shouldn't have
|
||||
* improved the ability of the manual file exporter process to report errors on a very large export that encounters errors after the dialog has closed
|
||||
* fixed the 'remember last used default tag service in manage tag dialogs' and its accompanying dropdown not saving their current value on options dialog ok. sorry for the trouble!
|
||||
* fixed the system that truncates very long filenames (for export folders and drag and drop exports) on Linux when the exporter is also outputting a sidecar that has a long extra suffix
|
||||
* the 'find potential duplicate pairs' routine that runs in idle time now properly obeys the work/rest times in `options->maintenance and processing`. previously, it was just the 'run now' routine that was resting in that way, and the idle thing was just doing a hardcoded 'work for 60 seconds every 10 mins or so'. thanks to the reporting user who cleverly noticed this
|
||||
* the `options->connection` page now mentions your proxy needs to be `http://`
|
||||
|
||||
### boring stuff
|
||||
|
||||
* updated the windows setup_venv.bat to allow for custom python or venv locations using parameters. this was so I could set up a multi-python testing situation easier
|
||||
* added some unit tests for the new URL encoding gubbins
|
||||
* improved un-encoded URL parsing in the downloader when the URL is relative and needs to be joined to the source url
|
||||
* improved some URL parsing and ingestion to better handle urls with non-ascii characters in the domain
|
||||
* replaced several 'does it start with "http"?' areas with a better and unified scheme/netloc test
|
||||
* wrote a routine to split URL paths into path components, and spammed it everywhere so this code is now unified. I expect we'll get a `PathComponent` class at some point, too. there will be a future question about what to do with double slashes, `//` in paths--it turns out the logic has been mixed here, and I think I will probably collapse them to `/` in all cases
|
||||
* rewrote an unhealthy call that indirectly caused the above multi-file post parsing problem
|
||||
* fixed some None/0 `NumberTest` stuff if you manage to enter '<0' or >-5 and similar
|
||||
* I figured out the problems with PyInstaller 6.x and some other stuff, there should be a 'Future Build' alongside this release in github for advanced users to test with
|
||||
|
||||
## [Version 568](https://github.com/hydrusnetwork/hydrus/releases/tag/v568)
|
||||
|
||||
### user contributions
|
||||
|
|
|
|||
|
|
@ -1783,7 +1783,7 @@ Response:
|
|||
"has_transparency" : false,
|
||||
"known_urls" : [
|
||||
"https://gelbooru.com/index.php?page=post&s=view&id=4841557",
|
||||
"https://img2.gelbooru.com//images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg",
|
||||
"https://img2.gelbooru.com/images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg",
|
||||
"http://origin-orig.deviantart.net/ed31/f/2019/210/7/8/beachqueen_samus_by_dandonfuga-ddcu1xg.jpg"
|
||||
],
|
||||
"ratings" : {
|
||||
|
|
@ -1971,7 +1971,7 @@ If you add `detailed_url_information=true`, a new entry, `detailed_known_urls`,
|
|||
"can_parse": true
|
||||
},
|
||||
{
|
||||
"normalised_url": "https://img2.gelbooru.com//images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg",
|
||||
"normalised_url": "https://img2.gelbooru.com/images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg",
|
||||
"url_type": 5,
|
||||
"url_type_string": "unknown url",
|
||||
"match_name": "unknown url",
|
||||
|
|
|
|||
|
|
@ -34,6 +34,41 @@
|
|||
<div class="content">
|
||||
<h1 id="changelog"><a href="#changelog">changelog</a></h1>
|
||||
<ul>
|
||||
<li>
|
||||
<h2 id="version_569"><a href="#version_569">version 569</a></h2>
|
||||
<ul>
|
||||
<li><h3>user contributions</h3></li>
|
||||
<li>thanks to a user, fixed a problem with the recent URL changes that caused downloaders examining multi-file posts to only grab the first file</li>
|
||||
<li>thanks to a user, all the menubar commands that launch a modal dialog are now suffix'd by an ellipsis</li>
|
||||
<li>thanks to a user, fixed an issue regarding KDE 6 quitting the program as soon as the pre-boot 'your database is missing a location, let's find it' repair dialog was ok'd</li>
|
||||
<li>thanks to a user, the application icon is fixed in KDE Plasma Wayland (and anything else that pulls icon from .desktop file). if you have been using a hydrus.desktop file and don't see a program icon, you should rename it to `/usr/share/applications/io.github.hydrusnetwork.hydrus.desktop` . more importantly, if you manage a package for hydrus--please output to this file path instead of `hydrus.desktop` if you make one</li>
|
||||
<li>thanks to a user, updated the `hydrus_client.sh` file to include `"$@"`, which passes parameters given to the .sh file to the .py call</li>
|
||||
<li><h3>more on last week's URL work</h3></li>
|
||||
<li>fixed the 'show the Request URL under "additional urls" submenu' thing on the file log list menu. I screwed up the logic and was effectively testing for when `1 != 1`</li>
|
||||
<li>the converter that generates a Referral URL now operates on the API/redirect conversion principle too--it normalises the Source URL to its 'Request URL' state--keeping defined ephemeral params and filling in defaults but dropping any extra gubbins not asked for--before applying the conversion</li>
|
||||
<li>fixed the 'manage url class' dialog to correctly display an example API/redirect-converted URL based on the new _request url_, not the _normalised url_ (so the api/redirect example will now show the new ephemeral params properly). this was working in requests correctly behind the scenes, it was just the example text box in the dialog that was showing wrong</li>
|
||||
<li>improved the 'is this query text pre-encoded?' test to check for `%hh`, where `h` is a hexadecimal character, instead of the hackier 'is % in it while not followed by whitespace or end of string?'</li>
|
||||
<li>improved/simplified/optimised the overall procedure that figures out if an entered URL is pre-encoded or not. this routine now only runs at the stage where a URL is ingested and it obeys the `%hh` rule. these ingestion points are currently: the text boxes in a urls downloader/simple downloader page; the 'import new sources' function of file log menus; a URL `ContentParser` in the parsing system; the test box in `manage url classes`; and the main gui's 'import url' landing pad, which is used by the drag and drop system, the clipboard watcher, and the client api's 'import url' command. note that this does not occur on 'manage known urls' editing, where you can do what you want with whatever, and I won't coerce it to anything</li>
|
||||
<li><h3>misc</h3></li>
|
||||
<li>fixed a variety of logical cases around >0, =0, !=0, <0 for the `NumberTest` objects I recently applied to system:duration and elsewhere. when it comes to file searching, files that have 'None' duration are now considered equivalent to files that have an explicit 0 duration in all cases. previously, I was trying to thread a needle where '=0' would find null results but <x would not, and it was a mess. now it all works the same way. if you want to search for 'duration < x' and want to exclude still images, either add a filetype pred or slap on 'has duration'</li>
|
||||
<li>improved the stability of the manual file exporter process. it was consulting an object in a thread that it shouldn't have</li>
|
||||
<li>improved the ability of the manual file exporter process to report errors on a very large export that encounters errors after the dialog has closed</li>
|
||||
<li>fixed the 'remember last used default tag service in manage tag dialogs' and its accompanying dropdown not saving their current value on options dialog ok. sorry for the trouble!</li>
|
||||
<li>fixed the system that truncates very long filenames (for export folders and drag and drop exports) on Linux when the exporter is also outputting a sidecar that has a long extra suffix</li>
|
||||
<li>the 'find potential duplicate pairs' routine that runs in idle time now properly obeys the work/rest times in `options->maintenance and processing`. previously, it was just the 'run now' routine that was resting in that way, and the idle thing was just doing a hardcoded 'work for 60 seconds every 10 mins or so'. thanks to the reporting user who cleverly noticed this</li>
|
||||
<li>the `options->connection` page now mentions your proxy needs to be `http://`</li>
|
||||
<li><h3>boring stuff</h3></li>
|
||||
<li>updated the windows setup_venv.bat to allow for custom python or venv locations using parameters. this was so I could set up a multi-python testing situation easier</li>
|
||||
<li>added some unit tests for the new URL encoding gubbins</li>
|
||||
<li>improved un-encoded URL parsing in the downloader when the URL is relative and needs to be joined to the source url</li>
|
||||
<li>improved some URL parsing and ingestion to better handle urls with non-ascii characters in the domain</li>
|
||||
<li>replaced several 'does it start with "http"?' areas with a better and unified scheme/netloc test</li>
|
||||
<li>wrote a routine to split URL paths into path components, and spammed it everywhere so this code is now unified. I expect we'll get a `PathComponent` class at some point, too. there will be a future question about what to do with double slashes, `//` in paths--it turns out the logic has been mixed here, and I think I will probably collapse them to `/` in all cases</li>
|
||||
<li>rewrote an unhealthy call that indirectly caused the above multi-file post parsing problem</li>
|
||||
<li>fixed some None/0 `NumberTest` stuff if you manage to enter '<0' or >-5 and similar</li>
|
||||
<li>I figured out the problems with PyInstaller 6.x and some other stuff, there should be a 'Future Build' alongside this release in github for advanced users to test with</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<h2 id="version_568"><a href="#version_568">version 568</a></h2>
|
||||
<p><i>Version 567 was cancelled, its changes folded into 568</i></p>
|
||||
|
|
|
|||
|
|
@ -105,7 +105,11 @@ class App( QW.QApplication ):
|
|||
self._pubsub = pubsub
|
||||
|
||||
self.setApplicationName( 'Hydrus Client' )
|
||||
self.setDesktopFileName( 'io.github.hydrusnetwork.hydrus' )
|
||||
|
||||
if HC.PLATFORM_LINUX:
|
||||
|
||||
self.setDesktopFileName( 'io.github.hydrusnetwork.hydrus' )
|
||||
|
||||
|
||||
self.setApplicationVersion( str( HC.SOFTWARE_VERSION ) )
|
||||
|
||||
|
|
@ -1426,20 +1430,52 @@ class Controller( ClientControllerInterface.ClientControllerInterface, HydrusCon
|
|||
|
||||
if self.new_options.GetBoolean( 'maintain_similar_files_duplicate_pairs_during_idle' ):
|
||||
|
||||
search_distance = self.new_options.GetInteger( 'similar_files_duplicate_pairs_search_distance' )
|
||||
work_done = False
|
||||
|
||||
search_stop_time = stop_time
|
||||
still_work_to_do = True
|
||||
|
||||
if search_stop_time is None:
|
||||
while still_work_to_do:
|
||||
|
||||
search_stop_time = HydrusTime.GetNow() + 60
|
||||
search_distance = CG.client_controller.new_options.GetInteger( 'similar_files_duplicate_pairs_search_distance' )
|
||||
|
||||
start_time = HydrusTime.GetNowPrecise()
|
||||
|
||||
work_time_ms = CG.client_controller.new_options.GetInteger( 'potential_duplicates_search_work_time_ms' )
|
||||
|
||||
work_time = work_time_ms / 1000
|
||||
|
||||
( still_work_to_do, num_done ) = CG.client_controller.WriteSynchronous( 'maintain_similar_files_search_for_potential_duplicates', search_distance, maintenance_mode = maintenance_mode, work_time_float = work_time )
|
||||
|
||||
if num_done > 0:
|
||||
|
||||
work_done = True
|
||||
|
||||
|
||||
if not still_work_to_do:
|
||||
|
||||
break
|
||||
|
||||
|
||||
if self.ShouldStopThisWork( maintenance_mode, stop_time = stop_time ):
|
||||
|
||||
break
|
||||
|
||||
|
||||
time_it_took = HydrusTime.GetNowPrecise() - start_time
|
||||
|
||||
rest_ratio = CG.client_controller.new_options.GetInteger( 'potential_duplicates_search_rest_percentage' ) / 100
|
||||
|
||||
reasonable_work_time = min( 5 * work_time, time_it_took )
|
||||
|
||||
time.sleep( reasonable_work_time * rest_ratio )
|
||||
|
||||
|
||||
self.WriteSynchronous( 'maintain_similar_files_search_for_potential_duplicates', search_distance, maintenance_mode = maintenance_mode, stop_time = search_stop_time )
|
||||
|
||||
from hydrus.client import ClientDuplicates
|
||||
|
||||
ClientDuplicates.DuplicatesManager.instance().RefreshMaintenanceNumbers()
|
||||
if work_done:
|
||||
|
||||
from hydrus.client import ClientDuplicates
|
||||
|
||||
ClientDuplicates.DuplicatesManager.instance().RefreshMaintenanceNumbers()
|
||||
|
||||
|
||||
|
||||
if self.ShouldStopThisWork( maintenance_mode, stop_time = stop_time ):
|
||||
|
|
|
|||
|
|
@ -2335,12 +2335,12 @@ class ContentParser( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
base_url = parsing_context[ 'url' ]
|
||||
|
||||
def clean_url( u ):
|
||||
def remove_pre_url_gubbins( u ):
|
||||
|
||||
# clears up when a source field starts with gubbins for some reason. e.g.:
|
||||
# (jap characters).avi | ranken [pixiv] http:/www.pixiv.net/member_illust.php?illust_id=48114073&mode=medium
|
||||
# (jap characters).avi | ranken [pixiv] http://www.pixiv.net/member_illust.php?illust_id=48114073&mode=medium
|
||||
# ->
|
||||
# http:/www.pixiv.net/member_illust.php?illust_id=48114073&mode=medium
|
||||
# http://www.pixiv.net/member_illust.php?illust_id=48114073&mode=medium
|
||||
|
||||
gumpf_until_scheme = r'^.*\s(?P<scheme>https?://)'
|
||||
|
||||
|
|
@ -2368,33 +2368,33 @@ class ContentParser( HydrusSerialisable.SerialisableBase ):
|
|||
return u
|
||||
|
||||
|
||||
clean_parsed_texts = []
|
||||
clean_urls = []
|
||||
|
||||
for parsed_text in parsed_texts:
|
||||
for unclean_url in parsed_texts:
|
||||
|
||||
if not ClientNetworkingFunctions.LooksLikeAFullURL( parsed_text ) and ( 'http://' in parsed_text or 'https://' in parsed_text ):
|
||||
if not ClientNetworkingFunctions.LooksLikeAFullURL( unclean_url ) and ( 'http://' in unclean_url or 'https://' in unclean_url ):
|
||||
|
||||
parsed_text = clean_url( parsed_text )
|
||||
unclean_url = remove_pre_url_gubbins( unclean_url )
|
||||
|
||||
|
||||
clean_parsed_texts.append( parsed_text )
|
||||
if not ClientNetworkingFunctions.LooksLikeAFullURL( unclean_url ):
|
||||
|
||||
try:
|
||||
|
||||
unclean_url = urllib.parse.urljoin( base_url, unclean_url )
|
||||
|
||||
except:
|
||||
|
||||
pass # welp
|
||||
|
||||
|
||||
|
||||
clean_url = ClientNetworkingFunctions.WashURL( unclean_url )
|
||||
|
||||
clean_urls.append( clean_url )
|
||||
|
||||
|
||||
parsed_texts = []
|
||||
|
||||
for clean_parsed_text in clean_parsed_texts:
|
||||
|
||||
try:
|
||||
|
||||
parsed_text = urllib.parse.urljoin( base_url, clean_parsed_text )
|
||||
|
||||
except:
|
||||
|
||||
parsed_text = clean_parsed_text
|
||||
|
||||
|
||||
parsed_texts.append( parsed_text )
|
||||
|
||||
parsed_texts = clean_urls
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -154,9 +154,9 @@ def GenerateExportFilename( destination_directory, media, terms, file_index, do_
|
|||
|
||||
|
||||
# sidecar suffixes, and in general we don't want to spam giganto strings to people's hard drives
|
||||
extra_characters_and_padding = 64
|
||||
extra_characters_and_padding = 64 + len( ext )
|
||||
|
||||
filename = HydrusPaths.ElideFilenameOrDirectorySafely( filename, num_characters_used_in_other_components = destination_directory_num_characters_in_filesystem + extra_characters_and_padding )
|
||||
filename = HydrusPaths.ElideFilenameOrDirectorySafely( filename, num_characters_already_used_in_this_component = extra_characters_and_padding, num_characters_used_in_other_components = destination_directory_num_characters_in_filesystem )
|
||||
|
||||
if do_not_use_filenames is not None:
|
||||
|
||||
|
|
|
|||
|
|
@ -98,6 +98,7 @@ from hydrus.client.interfaces import ClientControllerInterface
|
|||
from hydrus.client.media import ClientMediaResult
|
||||
from hydrus.client.metadata import ClientContentUpdates
|
||||
from hydrus.client.metadata import ClientTags
|
||||
from hydrus.client.networking import ClientNetworkingFunctions
|
||||
|
||||
MENU_ORDER = [ 'file', 'undo', 'pages', 'database', 'network', 'services', 'tags', 'pending', 'help' ]
|
||||
|
||||
|
|
@ -2192,7 +2193,7 @@ class FrameGUI( CAC.ApplicationCommandProcessorMixin, ClientGUITopLevelWindows.M
|
|||
|
||||
def _ImportURL(
|
||||
self,
|
||||
url,
|
||||
unclean_url,
|
||||
filterable_tags = None,
|
||||
additional_service_keys_to_tags = None,
|
||||
destination_page_name = None,
|
||||
|
|
@ -2213,6 +2214,8 @@ class FrameGUI( CAC.ApplicationCommandProcessorMixin, ClientGUITopLevelWindows.M
|
|||
additional_service_keys_to_tags = ClientTags.ServiceKeysToTags()
|
||||
|
||||
|
||||
url = ClientNetworkingFunctions.WashURL( unclean_url )
|
||||
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, for_server = True )
|
||||
|
||||
( url_type, match_name, can_parse, cannot_parse_reason ) = self._controller.network_engine.domain_manager.GetURLParseCapability( url )
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ from hydrus.client.gui.lists import ClientGUIListCtrl
|
|||
from hydrus.client.gui.widgets import ClientGUICommon
|
||||
from hydrus.client.gui.widgets import ClientGUIMenuButton
|
||||
from hydrus.client.networking import ClientNetworkingDomain
|
||||
from hydrus.client.networking import ClientNetworkingFunctions
|
||||
from hydrus.client.networking import ClientNetworkingGUG
|
||||
from hydrus.client.networking import ClientNetworkingURLClass
|
||||
|
||||
|
|
@ -2055,12 +2056,16 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
self._example_url_classes.setText( 'Example matches ok!' )
|
||||
self._example_url_classes.setObjectName( 'HydrusValid' )
|
||||
|
||||
for_server_normalised = url_class.Normalise( example_url, for_server = True )
|
||||
|
||||
self._for_server_normalised_url.setText( for_server_normalised )
|
||||
|
||||
normalised = url_class.Normalise( example_url )
|
||||
|
||||
self._normalised_url.setText( normalised )
|
||||
|
||||
self._referral_url_converter.SetExampleString( normalised )
|
||||
self._api_lookup_converter.SetExampleString( normalised )
|
||||
self._referral_url_converter.SetExampleString( for_server_normalised )
|
||||
self._api_lookup_converter.SetExampleString( for_server_normalised )
|
||||
|
||||
if url_class.UsesAPIURL():
|
||||
|
||||
|
|
@ -2106,15 +2111,11 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
|
||||
|
||||
for_server_normalised = url_class.Normalise( example_url, for_server = True )
|
||||
|
||||
self._for_server_normalised_url.setText( for_server_normalised )
|
||||
|
||||
try:
|
||||
|
||||
if url_class.UsesAPIURL():
|
||||
|
||||
api_lookup_url = url_class.GetAPIURL( normalised )
|
||||
api_lookup_url = url_class.GetAPIURL( example_url )
|
||||
|
||||
if url_class.Matches( api_lookup_url ):
|
||||
|
||||
|
|
@ -2450,14 +2451,16 @@ class EditURLClassesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
def _UpdateURLClassCheckerText( self ):
|
||||
|
||||
url = self._url_class_checker.text()
|
||||
unclean_url = self._url_class_checker.text()
|
||||
|
||||
if url == '':
|
||||
if unclean_url == '':
|
||||
|
||||
text = '<-- Enter a URL here to see which url class it currently matches!'
|
||||
|
||||
else:
|
||||
|
||||
url = ClientNetworkingFunctions.WashURL( unclean_url )
|
||||
|
||||
url_classes = self.GetValue()
|
||||
|
||||
domain_manager = ClientNetworkingDomain.NetworkDomainManager()
|
||||
|
|
|
|||
|
|
@ -1,13 +1,10 @@
|
|||
import os
|
||||
|
||||
from qtpy import QtCore as QC
|
||||
from qtpy import QtWidgets as QW
|
||||
from qtpy import QtGui as QG
|
||||
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusText
|
||||
|
||||
|
|
@ -70,6 +67,8 @@ def GetSourcesFromSourcesString( sources_string ):
|
|||
|
||||
sources = HydrusText.DeserialiseNewlinedTexts( sources_string )
|
||||
|
||||
sources = [ ClientNetworkingFunctions.WashURL( source ) for source in sources ]
|
||||
|
||||
return sources
|
||||
|
||||
def ExportToClipboard( file_seed_cache: ClientImportFileSeeds.FileSeedCache ):
|
||||
|
|
@ -528,14 +527,13 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
if selected_file_seed.IsURLFileImport():
|
||||
|
||||
main_url = selected_file_seed.file_seed_data
|
||||
request_url = selected_file_seed.file_seed_data
|
||||
normalised_url = selected_file_seed.file_seed_data_for_comparison
|
||||
referral_url = selected_file_seed.GetReferralURL()
|
||||
primary_urls = sorted( selected_file_seed.GetPrimaryURLs() )
|
||||
source_urls = sorted( selected_file_seed.GetSourceURLs() )
|
||||
|
||||
for_server_url = CG.client_controller.network_engine.domain_manager.GetURLToFetch( main_url )
|
||||
|
||||
nothing_interesting_going_on = main_url == for_server_url and referral_url is None and len( primary_urls ) == 0 and len( source_urls ) == 0
|
||||
nothing_interesting_going_on = request_url == normalised_url and referral_url is None and len( primary_urls ) == 0 and len( source_urls ) == 0
|
||||
|
||||
if nothing_interesting_going_on:
|
||||
|
||||
|
|
@ -545,9 +543,9 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
url_submenu = ClientGUIMenus.GenerateMenu( menu )
|
||||
|
||||
if main_url != for_server_url:
|
||||
if request_url != normalised_url:
|
||||
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, f'request url: {for_server_url}', copy_text = for_server_url )
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, f'request url: {request_url}', copy_text = request_url )
|
||||
|
||||
|
||||
if referral_url is not None:
|
||||
|
|
|
|||
|
|
@ -393,20 +393,22 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
|
|||
|
||||
general.Add( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
|
||||
|
||||
text = 'Enter strings such as "http://ip:port" or "http://user:pass@ip:port" to use for http and https traffic. It should take effect immediately on dialog ok.'
|
||||
text += os.linesep * 2
|
||||
text += 'NO PROXY DOES NOT WORK UNLESS YOU HAVE A CUSTOM BUILD OF REQUESTS, SORRY! no_proxy takes the form of comma-separated hosts/domains, just as in curl or the NO_PROXY environment variable. When http and/or https proxies are set, they will not be used for these.'
|
||||
text += os.linesep * 2
|
||||
text = 'PROTIP: Use a system-wide VPN or other software to handle this externally if you can. This tech is old and imperfect.'
|
||||
text += '\n' * 2
|
||||
text += 'Enter strings such as "http://ip:port" or "http://user:pass@ip:port" to use for http and https traffic. It should take effect immediately on dialog ok. Note that you have to enter "http://", not "https://" (an HTTP proxy forwards your traffic, which when you talk to an https:// address will be encrypted, but it does not wrap that in an extra layer of encryption itself).'
|
||||
text += '\n' * 2
|
||||
text += '"NO_PROXY" DOES NOT WORK UNLESS YOU HAVE A CUSTOM BUILD OF REQUESTS, SORRY! no_proxy takes the form of comma-separated hosts/domains, just as in curl or the NO_PROXY environment variable. When http and/or https proxies are set, they will not be used for these.'
|
||||
text += '\n' * 2
|
||||
|
||||
if ClientNetworkingSessions.SOCKS_PROXY_OK:
|
||||
|
||||
text += 'It looks like you have socks support! You should also be able to enter (socks4 or) "socks5://ip:port".'
|
||||
text += os.linesep
|
||||
text += 'It looks like you have SOCKS support! You should also be able to enter (socks4 or) "socks5://ip:port".'
|
||||
text += '\n'
|
||||
text += 'Use socks4a or socks5h to force remote DNS resolution, on the proxy server.'
|
||||
|
||||
else:
|
||||
|
||||
text += 'It does not look like you have socks support! If you want it, try adding "pysocks" (or "requests[socks]")!'
|
||||
text += 'It does not look like you have SOCKS support! If you want it, try adding "pysocks" (or "requests[socks]")!'
|
||||
|
||||
|
||||
st = ClientGUICommon.BetterStaticText( proxy_panel, text )
|
||||
|
|
@ -3215,6 +3217,9 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
|
|||
self._new_options.SetInteger( 'ac_read_list_height_num_chars', self._ac_read_list_height_num_chars.value() )
|
||||
self._new_options.SetInteger( 'ac_write_list_height_num_chars', self._ac_write_list_height_num_chars.value() )
|
||||
|
||||
self._new_options.SetBoolean( 'save_default_tag_service_tab_on_change', self._save_default_tag_service_tab_on_change.isChecked() )
|
||||
self._new_options.SetKey( 'default_tag_service_tab', self._default_tag_service_tab.GetValue() )
|
||||
|
||||
self._new_options.SetBoolean( 'always_show_system_everything', self._always_show_system_everything.isChecked() )
|
||||
self._new_options.SetBoolean( 'filter_inbox_and_archive_predicates', self._filter_inbox_and_archive_predicates.isChecked() )
|
||||
|
||||
|
|
|
|||
|
|
@ -806,7 +806,7 @@ class ReviewExportFilesPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
|
||||
|
||||
def do_it( directory, metadata_routers, delete_afterwards, export_symlinks, quit_afterwards ):
|
||||
def do_it( directory, metadata_routers, delete_afterwards, export_symlinks, quit_afterwards, media_to_number_indices ):
|
||||
|
||||
job_status = ClientThreading.JobStatus( cancellable = True )
|
||||
|
||||
|
|
@ -818,7 +818,7 @@ class ReviewExportFilesPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
for ( index, ( media, dest_path ) ) in enumerate( to_do ):
|
||||
|
||||
number = self._media_to_number_indices[ media ]
|
||||
number = media_to_number_indices[ media ]
|
||||
|
||||
if job_status.IsCancelled():
|
||||
|
||||
|
|
@ -881,7 +881,16 @@ class ReviewExportFilesPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
except:
|
||||
|
||||
ClientGUIDialogsMessage.ShowCritical( self, 'Problem during file export!', f'Encountered a problem while attempting to export file #{HydrusData.ToHumanInt( number )}:\n\n{traceback.format_exc()}' )
|
||||
if QP.isValid( self ):
|
||||
|
||||
win = self
|
||||
|
||||
else:
|
||||
|
||||
win = CG.client_controller.gui
|
||||
|
||||
|
||||
ClientGUIDialogsMessage.ShowCritical( win, 'Problem during file export!', f'Encountered a problem while attempting to export file #{HydrusData.ToHumanInt( number )}:\n\n{traceback.format_exc()}' )
|
||||
|
||||
break
|
||||
|
||||
|
|
@ -938,7 +947,7 @@ class ReviewExportFilesPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
QP.CallAfter( qt_done, quit_afterwards )
|
||||
|
||||
|
||||
CG.client_controller.CallToThread( do_it, directory, metadata_routers, delete_afterwards, export_symlinks, quit_afterwards )
|
||||
CG.client_controller.CallToThread( do_it, directory, metadata_routers, delete_afterwards, export_symlinks, quit_afterwards, self._media_to_number_indices )
|
||||
|
||||
|
||||
def _GetPath( self, media ):
|
||||
|
|
|
|||
|
|
@ -64,6 +64,7 @@ from hydrus.client.importing.options import PresentationImportOptions
|
|||
from hydrus.client.media import ClientMedia
|
||||
from hydrus.client.metadata import ClientContentUpdates
|
||||
from hydrus.client.metadata import ClientTags
|
||||
from hydrus.client.networking import ClientNetworkingFunctions
|
||||
from hydrus.client.search import ClientSearch
|
||||
|
||||
management_panel_types_to_classes = {}
|
||||
|
|
@ -3443,9 +3444,9 @@ class ManagementPanelImporterSimpleDownloader( ManagementPanelImporter ):
|
|||
self._RefreshFormulae()
|
||||
|
||||
|
||||
def _PendPageURLs( self, urls ):
|
||||
def _PendPageURLs( self, unclean_urls ):
|
||||
|
||||
urls = [ url for url in urls if url.startswith( 'http' ) ]
|
||||
urls = [ ClientNetworkingFunctions.WashURL( unclean_url ) for unclean_url in unclean_urls if ClientNetworkingFunctions.LooksLikeAFullURL( unclean_url ) ]
|
||||
|
||||
simple_downloader_formula = self._formulae.GetValue()
|
||||
|
||||
|
|
@ -3757,7 +3758,7 @@ class ManagementPanelImporterURLs( ManagementPanelImporter ):
|
|||
self._import_options_button.tagImportOptionsChanged.connect( self._urls_import.SetTagImportOptions )
|
||||
|
||||
|
||||
def _PendURLs( self, urls, filterable_tags = None, additional_service_keys_to_tags = None ):
|
||||
def _PendURLs( self, unclean_urls, filterable_tags = None, additional_service_keys_to_tags = None ):
|
||||
|
||||
if filterable_tags is None:
|
||||
|
||||
|
|
@ -3769,7 +3770,7 @@ class ManagementPanelImporterURLs( ManagementPanelImporter ):
|
|||
additional_service_keys_to_tags = ClientTags.ServiceKeysToTags()
|
||||
|
||||
|
||||
urls = [ url for url in urls if url.startswith( 'http' ) ]
|
||||
urls = [ ClientNetworkingFunctions.WashURL( unclean_url ) for unclean_url in unclean_urls if ClientNetworkingFunctions.LooksLikeAFullURL( unclean_url ) ]
|
||||
|
||||
self._urls_import.PendURLs( urls, filterable_tags = filterable_tags, additional_service_keys_to_tags = additional_service_keys_to_tags )
|
||||
|
||||
|
|
|
|||
|
|
@ -133,17 +133,7 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
|
|||
self.file_seed_data = file_seed_data
|
||||
self.file_seed_data_for_comparison = file_seed_data
|
||||
|
||||
if self.file_seed_type == FILE_SEED_TYPE_URL:
|
||||
|
||||
try:
|
||||
|
||||
self.file_seed_data_for_comparison = CG.client_controller.network_engine.domain_manager.NormaliseURL( self.file_seed_data )
|
||||
|
||||
except:
|
||||
|
||||
pass
|
||||
|
||||
|
||||
self.Normalise() # this fixes the comparison file seed data and fails safely
|
||||
|
||||
self.created = HydrusTime.GetNow()
|
||||
self.modified = self.created
|
||||
|
|
@ -1260,6 +1250,14 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
|
||||
|
||||
def SetFileSeedData( self, file_seed_data: str ):
|
||||
|
||||
self.file_seed_data = file_seed_data
|
||||
self.file_seed_data_for_comparison = file_seed_data
|
||||
|
||||
self.Normalise() # this fixes the comparison file seed data and fails safely
|
||||
|
||||
|
||||
def SetHash( self, hash ):
|
||||
|
||||
if hash is not None:
|
||||
|
|
@ -1588,9 +1586,7 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
duplicate_file_seed = self.Duplicate() # inherits all urls and tags from here
|
||||
|
||||
duplicate_file_seed.file_seed_data = child_url
|
||||
|
||||
duplicate_file_seed.Normalise()
|
||||
duplicate_file_seed.SetFileSeedData( child_url )
|
||||
|
||||
duplicate_file_seed.SetReferralURL( url_for_child_referral )
|
||||
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ from hydrus.client import ClientTime
|
|||
from hydrus.client.importing.options import NoteImportOptions
|
||||
from hydrus.client.metadata import ClientContentUpdates
|
||||
from hydrus.client.metadata import ClientMetadataMigrationCore
|
||||
from hydrus.client.networking import ClientNetworkingFunctions
|
||||
|
||||
class SingleFileMetadataExporter( ClientMetadataMigrationCore.ImporterExporterNode ):
|
||||
|
||||
|
|
@ -406,7 +407,9 @@ class SingleFileMetadataExporterMediaURLs( SingleFileMetadataExporterMedia, Hydr
|
|||
|
||||
try:
|
||||
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( row )
|
||||
url = ClientNetworkingFunctions.WashURL( row )
|
||||
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url )
|
||||
|
||||
urls.append( url )
|
||||
|
||||
|
|
|
|||
|
|
@ -1518,24 +1518,8 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
if url_class is None:
|
||||
|
||||
p = ClientNetworkingFunctions.ParseURL( url )
|
||||
|
||||
scheme = p.scheme
|
||||
netloc = p.netloc
|
||||
path = p.path
|
||||
params = p.params
|
||||
|
||||
# this puts them all in alphabetical order
|
||||
|
||||
( query_dict, single_value_parameters, param_order ) = ClientNetworkingFunctions.ConvertQueryTextToDict( p.query )
|
||||
|
||||
query = ClientNetworkingFunctions.ConvertQueryDictToText( query_dict, single_value_parameters )
|
||||
|
||||
fragment = ''
|
||||
|
||||
r = urllib.parse.ParseResult( scheme, netloc, path, params, query, fragment )
|
||||
|
||||
normalised_url = r.geturl()
|
||||
# this is less about washing as it is about stripping the fragment
|
||||
normalised_url = ClientNetworkingFunctions.WashURL( url, keep_fragment = False )
|
||||
|
||||
else:
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
import http.cookiejar
|
||||
import re
|
||||
import typing
|
||||
|
||||
import unicodedata
|
||||
import urllib.parse
|
||||
|
||||
|
|
@ -102,6 +104,27 @@ def ConvertHTTPToHTTPS( url ):
|
|||
|
||||
|
||||
|
||||
def ConvertPathTextToList( path: str ) -> typing.List[ str ]:
|
||||
|
||||
# yo sometimes you see a URL with double slashes in a weird place. maybe we should just split( '/' ) and then remove empty '' results?
|
||||
|
||||
# /post/show/1326143/akunim-anthro-armband-armwear-clothed-clothing-fem
|
||||
|
||||
# for a while we've had URLs like this:
|
||||
# https://img2.gelbooru.com//images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg
|
||||
# I was going to be careful as I unified all this to preserve the double-slash to help legacy known url storage matching, but it seems we've been nuking the extra slash for ages in actual db storage, so w/e!
|
||||
while path.startswith( '/' ):
|
||||
|
||||
path = path[ 1 : ]
|
||||
|
||||
|
||||
# post/show/1326143/akunim-anthro-armband-armwear-clothed-clothing-fem
|
||||
|
||||
path_components = path.split( '/' )
|
||||
|
||||
return path_components
|
||||
|
||||
|
||||
def ConvertQueryDictToText( query_dict, single_value_parameters, param_order = None ):
|
||||
|
||||
# we now do everything with requests, which does all the unicode -> %20 business naturally, phew
|
||||
|
|
@ -152,6 +175,7 @@ def ConvertQueryDictToText( query_dict, single_value_parameters, param_order = N
|
|||
|
||||
return query_text
|
||||
|
||||
|
||||
def ConvertQueryTextToDict( query_text ):
|
||||
|
||||
# in the old version of this func, we played silly games with character encoding. I made the foolish decision to try to handle/save URLs with %20 stuff decoded
|
||||
|
|
@ -187,11 +211,6 @@ def ConvertQueryTextToDict( query_text ):
|
|||
continue
|
||||
|
||||
|
||||
if '%' not in value:
|
||||
|
||||
value = urllib.parse.quote( value, safe = '+' )
|
||||
|
||||
|
||||
single_value_parameters.append( value )
|
||||
param_order.append( None )
|
||||
|
||||
|
|
@ -199,16 +218,6 @@ def ConvertQueryTextToDict( query_text ):
|
|||
|
||||
( key, value ) = result
|
||||
|
||||
if '%' not in key:
|
||||
|
||||
key = urllib.parse.quote( key, safe = '+' )
|
||||
|
||||
|
||||
if '%' not in value:
|
||||
|
||||
value = urllib.parse.quote( value, safe = '+' )
|
||||
|
||||
|
||||
param_order.append( key )
|
||||
|
||||
query_dict[ key ] = value
|
||||
|
|
@ -218,6 +227,32 @@ def ConvertQueryTextToDict( query_text ):
|
|||
return ( query_dict, single_value_parameters, param_order )
|
||||
|
||||
|
||||
def EnsureURLInfoIsEncoded( path_components: typing.List[ str ], query_dict: typing.Dict[ str, str ], single_value_parameters: typing.List[ str ] ):
|
||||
|
||||
# ok so the user just posted a URL at us, and this query dict could either be from a real url, like "tags=skirt%20blonde_hair", or it could be a pretty URL they typed or whatever, "tags=skirt blonde_hair"
|
||||
# so, let's do our best to figure out if the thing was pre-encoded or not, and wash it through a safe encoding process so it is encoded when we give it back
|
||||
# what's the potential problem? '+' is a special character that may or may not be encoded, e.g. "tags=6%2Bgirls+skirt" WEW
|
||||
|
||||
percent_encoding_re = re.compile( r'%[0-9A-Fa-f]{2}' )
|
||||
|
||||
all_gubbins = set( path_components )
|
||||
all_gubbins.update( query_dict.keys() )
|
||||
all_gubbins.update( query_dict.values() )
|
||||
all_gubbins.update( single_value_parameters )
|
||||
|
||||
there_are_percent_encoding_chars = True in ( percent_encoding_re.search( text ) is not None for text in all_gubbins )
|
||||
|
||||
# if there are percent-encoded characters anywhere, we have to assume the whole URL is already encoded correctly!
|
||||
if not there_are_percent_encoding_chars:
|
||||
|
||||
path_components = [ urllib.parse.quote( value, safe = '+' ) for value in path_components ]
|
||||
query_dict = { urllib.parse.quote( key, safe = '+' ) : urllib.parse.quote( value, safe = '+' ) for ( key, value ) in query_dict.items() }
|
||||
single_value_parameters = [ urllib.parse.quote( value, safe = '+' ) for value in single_value_parameters ]
|
||||
|
||||
|
||||
return ( path_components, query_dict, single_value_parameters )
|
||||
|
||||
|
||||
def ConvertURLIntoDomain( url ):
|
||||
|
||||
parser_result = ParseURL( url )
|
||||
|
|
@ -372,14 +407,14 @@ def LooksLikeAFullURL( text: str ) -> bool:
|
|||
|
||||
try:
|
||||
|
||||
result = urllib.parse.urlparse( text )
|
||||
p = ParseURL( text )
|
||||
|
||||
if result.scheme == '':
|
||||
if p.scheme == '':
|
||||
|
||||
return False
|
||||
|
||||
|
||||
if result.netloc == '':
|
||||
if p.netloc == '':
|
||||
|
||||
return False
|
||||
|
||||
|
|
@ -414,6 +449,7 @@ def NormaliseAndFilterAssociableURLs( urls ):
|
|||
|
||||
return associable_urls
|
||||
|
||||
|
||||
def ParseURL( url: str ) -> urllib.parse.ParseResult:
|
||||
|
||||
url = url.strip()
|
||||
|
|
@ -430,6 +466,47 @@ def ParseURL( url: str ) -> urllib.parse.ParseResult:
|
|||
|
||||
|
||||
|
||||
def WashURL( url: str, keep_fragment = True ) -> str:
|
||||
|
||||
if not LooksLikeAFullURL( url ):
|
||||
|
||||
return url
|
||||
|
||||
|
||||
try:
|
||||
|
||||
p = ParseURL( url )
|
||||
|
||||
scheme = p.scheme
|
||||
netloc = p.netloc
|
||||
params = p.params # just so you know, this is ancient web semicolon tech, can be ignored
|
||||
fragment = p.fragment
|
||||
|
||||
path_components = ConvertPathTextToList( p.path )
|
||||
( query_dict, single_value_parameters, param_order ) = ConvertQueryTextToDict( p.query )
|
||||
|
||||
( path_components, query_dict, single_value_parameters ) = EnsureURLInfoIsEncoded( path_components, query_dict, single_value_parameters )
|
||||
|
||||
path = '/' + '/'.join( path_components )
|
||||
query = ConvertQueryDictToText( query_dict, single_value_parameters )
|
||||
|
||||
if not keep_fragment:
|
||||
|
||||
fragment = ''
|
||||
|
||||
|
||||
r = urllib.parse.ParseResult( scheme, netloc, path, params, query, fragment )
|
||||
|
||||
clean_url = r.geturl()
|
||||
|
||||
return clean_url
|
||||
|
||||
except:
|
||||
|
||||
return url
|
||||
|
||||
|
||||
|
||||
OH_NO_NO_NETLOC_CHARACTERS = '?#'
|
||||
OH_NO_NO_NETLOC_CHARACTERS_UNICODE_TRANSLATE = { ord( char ) : '_' for char in OH_NO_NO_NETLOC_CHARACTERS }
|
||||
|
||||
|
|
@ -442,6 +519,7 @@ def RemoveWWWFromDomain( domain ):
|
|||
|
||||
return domain
|
||||
|
||||
|
||||
def UnicodeNormaliseURL( url: str ):
|
||||
|
||||
if url.startswith( 'file:' ):
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import re
|
||||
import urllib.parse
|
||||
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
|
|
@ -108,12 +109,11 @@ class GalleryURLGenerator( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
raise HydrusExceptions.GUGException( 'Replacement phrase not in URL template!' )
|
||||
|
||||
|
||||
if '%' in query_text:
|
||||
if re.search( r'%[0-9A-Fa-f]{2}', query_text ) is not None:
|
||||
|
||||
# redundant test but leave it in for now
|
||||
if ' ' in query_text or '% ' in query_text or query_text.endswith( '%' ):
|
||||
if ' ' in query_text:
|
||||
|
||||
# there is probably a legit % character here that should be encoded
|
||||
# against probability, there is probably a legit % character here that should be encoded
|
||||
|
||||
search_terms = query_text.split( ' ' )
|
||||
|
||||
|
|
|
|||
|
|
@ -42,7 +42,9 @@ def ConvertURLClassesIntoAPIPairs( url_classes ):
|
|||
continue
|
||||
|
||||
|
||||
api_url = url_class.GetAPIURL( url_class.GetExampleURL() )
|
||||
example_url = url_class.GetExampleURL()
|
||||
|
||||
api_url = url_class.GetAPIURL( example_url )
|
||||
|
||||
for other_url_class in url_classes:
|
||||
|
||||
|
|
@ -387,19 +389,10 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return netloc
|
||||
|
||||
|
||||
def _ClipAndFleshOutPath( self, path: str, for_server: bool ):
|
||||
def _ClipAndFleshOutPath( self, path_components: typing.List[ str ], for_server: bool ):
|
||||
|
||||
# /post/show/1326143/akunim-anthro-armband-armwear-clothed-clothing-fem
|
||||
|
||||
while path.startswith( '/' ):
|
||||
|
||||
path = path[ 1 : ]
|
||||
|
||||
|
||||
# post/show/1326143/akunim-anthro-armband-armwear-clothed-clothing-fem
|
||||
|
||||
path_components = path.split( '/' )
|
||||
|
||||
do_clip = self.UsesAPIURL() or not for_server
|
||||
flesh_out = len( path_components ) < len( self._path_components )
|
||||
|
||||
|
|
@ -425,21 +418,17 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
clipped_path_components.append( clipped_path_component )
|
||||
|
||||
|
||||
path = '/'.join( clipped_path_components )
|
||||
path_components = clipped_path_components
|
||||
|
||||
|
||||
# post/show/1326143
|
||||
|
||||
path = '/' + path
|
||||
path = '/' + '/'.join( path_components )
|
||||
|
||||
# /post/show/1326143
|
||||
|
||||
return path
|
||||
|
||||
|
||||
def _ClipAndFleshOutQuery( self, query: str, for_server: bool ):
|
||||
|
||||
( query_dict, single_value_parameters, param_order ) = ClientNetworkingFunctions.ConvertQueryTextToDict( query )
|
||||
def _ClipAndFleshOutQuery( self, query_dict: typing.Dict[ str, str ], single_value_parameters: typing.List[ str ], param_order: typing.List[ str ], for_server: bool ):
|
||||
|
||||
query_dict_keys_to_parameters = {}
|
||||
|
||||
|
|
@ -1034,16 +1023,11 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return self._api_lookup_converter
|
||||
|
||||
|
||||
def GetAPIURL( self, url = None ):
|
||||
def GetAPIURL( self, url ):
|
||||
|
||||
if url is None:
|
||||
|
||||
url = self._example_url
|
||||
|
||||
request_url = self.Normalise( url, for_server = True )
|
||||
|
||||
url = self.Normalise( url, for_server = True )
|
||||
|
||||
return self._api_lookup_converter.Convert( url )
|
||||
return self._api_lookup_converter.Convert( request_url )
|
||||
|
||||
|
||||
def GetClassKey( self ):
|
||||
|
|
@ -1093,12 +1077,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
page_index_path_component_index = self._gallery_index_identifier
|
||||
|
||||
while path.startswith( '/' ):
|
||||
|
||||
path = path[ 1 : ]
|
||||
|
||||
|
||||
path_components = path.split( '/' )
|
||||
path_components = ClientNetworkingFunctions.ConvertPathTextToList( path )
|
||||
|
||||
try:
|
||||
|
||||
|
|
@ -1195,9 +1174,11 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
elif self._send_referral_url in ( SEND_REFERRAL_URL_CONVERTER_IF_NONE_PROVIDED, SEND_REFERRAL_URL_ONLY_CONVERTER ):
|
||||
|
||||
request_url = self.Normalise( url, for_server = True )
|
||||
|
||||
try:
|
||||
|
||||
converted_referral_url = self._referral_url_converter.Convert( url )
|
||||
converted_referral_url = self._referral_url_converter.Convert( request_url )
|
||||
|
||||
except HydrusExceptions.StringConvertException:
|
||||
|
||||
|
|
@ -1331,9 +1312,12 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
fragment = ''
|
||||
|
||||
|
||||
path_components = ClientNetworkingFunctions.ConvertPathTextToList( p.path )
|
||||
( query_dict, single_value_parameters, param_order ) = ClientNetworkingFunctions.ConvertQueryTextToDict( p.query )
|
||||
|
||||
netloc = self._ClipNetLoc( p.netloc )
|
||||
path = self._ClipAndFleshOutPath( p.path, for_server )
|
||||
query = self._ClipAndFleshOutQuery( p.query, for_server )
|
||||
path = self._ClipAndFleshOutPath( path_components, for_server )
|
||||
query = self._ClipAndFleshOutQuery( query_dict, single_value_parameters, param_order, for_server )
|
||||
|
||||
r = urllib.parse.ParseResult( scheme, netloc, path, params, query, fragment )
|
||||
|
||||
|
|
@ -1442,32 +1426,29 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
|
||||
|
||||
url_path = p.path
|
||||
path = p.path
|
||||
query = p.query
|
||||
|
||||
while url_path.startswith( '/' ):
|
||||
|
||||
url_path = url_path[ 1 : ]
|
||||
|
||||
|
||||
url_path_components = url_path.split( '/' )
|
||||
path_components = ClientNetworkingFunctions.ConvertPathTextToList( path )
|
||||
( query_dict, single_value_parameters, param_order ) = ClientNetworkingFunctions.ConvertQueryTextToDict( query )
|
||||
|
||||
if self._no_more_path_components_than_this:
|
||||
|
||||
if len( url_path_components ) > len( self._path_components ):
|
||||
if len( path_components ) > len( self._path_components ):
|
||||
|
||||
raise HydrusExceptions.URLClassException( '"{}" has {} path components, but I will not allow more than my defined {}!'.format( url_path, len( url_path_components ), len( self._path_components ) ) )
|
||||
raise HydrusExceptions.URLClassException( '"{}" has {} path components, but I will not allow more than my defined {}!'.format( path, len( path_components ), len( self._path_components ) ) )
|
||||
|
||||
|
||||
|
||||
for ( index, ( string_match, default ) ) in enumerate( self._path_components ):
|
||||
|
||||
if len( url_path_components ) > index:
|
||||
if len( path_components ) > index:
|
||||
|
||||
url_path_component = url_path_components[ index ]
|
||||
path_component = path_components[ index ]
|
||||
|
||||
try:
|
||||
|
||||
string_match.Test( url_path_component )
|
||||
string_match.Test( path_component )
|
||||
|
||||
except HydrusExceptions.StringMatchException as e:
|
||||
|
||||
|
|
@ -1478,24 +1459,22 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
if index + 1 == len( self._path_components ):
|
||||
|
||||
message = '"{}" has {} path components, but I was expecting {}!'.format( url_path, len( url_path_components ), len( self._path_components ) )
|
||||
message = '"{}" has {} path components, but I was expecting {}!'.format( path, len( path_components ), len( self._path_components ) )
|
||||
|
||||
else:
|
||||
|
||||
message = '"{}" has {} path components, but I was expecting at least {} and maybe as many as {}!'.format( url_path, len( url_path_components ), index + 1, len( self._path_components ) )
|
||||
message = '"{}" has {} path components, but I was expecting at least {} and maybe as many as {}!'.format( path, len( path_components ), index + 1, len( self._path_components ) )
|
||||
|
||||
|
||||
raise HydrusExceptions.URLClassException( message )
|
||||
|
||||
|
||||
|
||||
( url_query_dict, single_value_parameters, param_order ) = ClientNetworkingFunctions.ConvertQueryTextToDict( p.query )
|
||||
|
||||
if self._no_more_parameters_than_this:
|
||||
|
||||
good_fixed_names = { parameter.GetName() for parameter in self._parameters if isinstance( parameter, URLClassParameterFixedName ) }
|
||||
|
||||
for ( name, value ) in url_query_dict.items():
|
||||
for ( name, value ) in query_dict.items():
|
||||
|
||||
if name not in good_fixed_names:
|
||||
|
||||
|
|
@ -1510,7 +1489,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
name = parameter.GetName()
|
||||
|
||||
if name not in url_query_dict:
|
||||
if name not in query_dict:
|
||||
|
||||
if parameter.MustBeInOriginalURL():
|
||||
|
||||
|
|
@ -1522,7 +1501,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
|
||||
|
||||
value = url_query_dict[ name ]
|
||||
value = query_dict[ name ]
|
||||
|
||||
try:
|
||||
|
||||
|
|
@ -1537,7 +1516,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
if len( single_value_parameters ) > 0 and not self._has_single_value_parameters and self._no_more_parameters_than_this:
|
||||
|
||||
raise HydrusExceptions.URLClassException( '"{}" has unexpected single-value parameters, but I am set to not allow any unexpected parameters!'.format( url_path ) )
|
||||
raise HydrusExceptions.URLClassException( '"{}" has unexpected single-value parameters, but I am set to not allow any unexpected parameters!'.format( query ) )
|
||||
|
||||
|
||||
if self._has_single_value_parameters:
|
||||
|
|
|
|||
|
|
@ -393,11 +393,25 @@ class NumberTest( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
if self.operator == NUMBER_TEST_OPERATOR_LESS_THAN:
|
||||
|
||||
return lambda x: x is not None and x < self.value
|
||||
if self.value > 0:
|
||||
|
||||
return lambda x: x is None or x < self.value
|
||||
|
||||
else:
|
||||
|
||||
return lambda x: x is not None and x < self.value
|
||||
|
||||
|
||||
elif self.operator == NUMBER_TEST_OPERATOR_GREATER_THAN:
|
||||
|
||||
return lambda x: x is not None and x > self.value
|
||||
if self.value < 0:
|
||||
|
||||
return lambda x: x is None or x > self.value
|
||||
|
||||
else:
|
||||
|
||||
return lambda x: x is not None and x > self.value
|
||||
|
||||
|
||||
elif self.operator == NUMBER_TEST_OPERATOR_EQUAL:
|
||||
|
||||
|
|
@ -415,18 +429,39 @@ class NumberTest( HydrusSerialisable.SerialisableBase ):
|
|||
lower = self.value * ( 1 - self.extra_value )
|
||||
upper = self.value * ( 1 + self.extra_value )
|
||||
|
||||
return lambda x: x is not None and lower < x < upper
|
||||
if lower <= 0:
|
||||
|
||||
return lambda x: x is None or x < upper
|
||||
|
||||
else:
|
||||
|
||||
return lambda x: x is not None and lower < x < upper
|
||||
|
||||
|
||||
elif self.operator == NUMBER_TEST_OPERATOR_APPROXIMATE_ABSOLUTE:
|
||||
|
||||
lower = self.value - self.extra_value
|
||||
upper = self.value + self.extra_value
|
||||
|
||||
return lambda x: x is not None and lower < x < upper
|
||||
if lower <= 0:
|
||||
|
||||
return lambda x: x is None or x < upper
|
||||
|
||||
else:
|
||||
|
||||
return lambda x: x is not None and lower < x < upper
|
||||
|
||||
|
||||
elif self.operator == NUMBER_TEST_OPERATOR_NOT_EQUAL:
|
||||
|
||||
return lambda x: x is not None and x != self.value
|
||||
if self.value == 0:
|
||||
|
||||
return lambda x: x is not None and x != self.value
|
||||
|
||||
else:
|
||||
|
||||
return lambda x: x is None or x != self.value
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -434,17 +469,31 @@ class NumberTest( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
if self.operator == NUMBER_TEST_OPERATOR_LESS_THAN:
|
||||
|
||||
return [ f'{variable_name} < {self.value}' ]
|
||||
if self.value > 0:
|
||||
|
||||
return [ f'( {variable_name} IS NULL OR {variable_name} < {self.value} )' ]
|
||||
|
||||
else:
|
||||
|
||||
return [ f'{variable_name} < {self.value}' ]
|
||||
|
||||
|
||||
elif self.operator == NUMBER_TEST_OPERATOR_GREATER_THAN:
|
||||
|
||||
return [ f'{variable_name} > {self.value}' ]
|
||||
if self.value < 0:
|
||||
|
||||
return [ f'( {variable_name} IS NULL OR {variable_name} > {self.value} )' ]
|
||||
|
||||
else:
|
||||
|
||||
return [ f'{variable_name} > {self.value}' ]
|
||||
|
||||
|
||||
elif self.operator == NUMBER_TEST_OPERATOR_EQUAL:
|
||||
|
||||
if self.value == 0:
|
||||
|
||||
return [ f'({variable_name} IS NULL OR {variable_name} = {self.value})' ]
|
||||
return [ f'( {variable_name} IS NULL OR {variable_name} = {self.value} )' ]
|
||||
|
||||
else:
|
||||
|
||||
|
|
@ -456,18 +505,39 @@ class NumberTest( HydrusSerialisable.SerialisableBase ):
|
|||
lower = self.value * ( 1 - self.extra_value )
|
||||
upper = self.value * ( 1 + self.extra_value )
|
||||
|
||||
return [ f'{variable_name} > {lower}', f'{variable_name} < {upper}' ]
|
||||
if lower <= 0:
|
||||
|
||||
return [ f'( {variable_name} is NULL OR {variable_name} < {upper} )' ]
|
||||
|
||||
else:
|
||||
|
||||
return [ f'{variable_name} > {lower}', f'{variable_name} < {upper}' ]
|
||||
|
||||
|
||||
elif self.operator == NUMBER_TEST_OPERATOR_APPROXIMATE_ABSOLUTE:
|
||||
|
||||
lower = self.value - self.extra_value
|
||||
upper = self.value + self.extra_value
|
||||
|
||||
return [ f'{variable_name} > {lower}', f'{variable_name} < {upper}' ]
|
||||
if lower <= 0:
|
||||
|
||||
return [ f'( {variable_name} IS NULL OR {variable_name} < {upper} )' ]
|
||||
|
||||
else:
|
||||
|
||||
return [ f'{variable_name} > {lower}', f'{variable_name} < {upper}' ]
|
||||
|
||||
|
||||
elif self.operator == NUMBER_TEST_OPERATOR_NOT_EQUAL:
|
||||
|
||||
return [ f'{variable_name} != {self.value}' ]
|
||||
if self.value == 0:
|
||||
|
||||
return [ f'{variable_name} IS NOT NULL AND {variable_name} != {self.value}' ]
|
||||
|
||||
else:
|
||||
|
||||
return [ f'( {variable_name} IS NULL OR {variable_name} != {self.value} )' ]
|
||||
|
||||
|
||||
|
||||
return []
|
||||
|
|
|
|||
|
|
@ -105,7 +105,7 @@ options = {}
|
|||
# Misc
|
||||
|
||||
NETWORK_VERSION = 20
|
||||
SOFTWARE_VERSION = 568
|
||||
SOFTWARE_VERSION = 569
|
||||
CLIENT_API_VERSION = 63
|
||||
|
||||
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )
|
||||
|
|
|
|||
|
|
@ -5223,7 +5223,7 @@ class TestClientAPI( unittest.TestCase ):
|
|||
media_results = []
|
||||
file_info_managers = []
|
||||
|
||||
urls = { "https://gelbooru.com/index.php?page=post&s=view&id=4841557", "https://img2.gelbooru.com//images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg" }
|
||||
urls = { "https://gelbooru.com/index.php?page=post&s=view&id=4841557", "https://img2.gelbooru.com/images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg" }
|
||||
|
||||
sorted_urls = sorted( urls )
|
||||
|
||||
|
|
@ -5492,7 +5492,7 @@ class TestClientAPI( unittest.TestCase ):
|
|||
|
||||
detailed_known_urls_metadata_row[ 'detailed_known_urls' ] = [
|
||||
{'normalised_url' : 'https://gelbooru.com/index.php?id=4841557&page=post&s=view', 'url_type' : 0, 'url_type_string' : 'post url', 'match_name' : 'gelbooru file page', 'can_parse' : True},
|
||||
{'normalised_url' : 'https://img2.gelbooru.com//images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg', 'url_type' : 5, 'url_type_string' : 'unknown url', 'match_name' : 'unknown url', 'can_parse' : False, 'cannot_parse_reason' : 'unknown url class'}
|
||||
{'normalised_url' : 'https://img2.gelbooru.com/images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg', 'url_type' : 5, 'url_type_string' : 'unknown url', 'match_name' : 'unknown url', 'can_parse' : False, 'cannot_parse_reason' : 'unknown url class'}
|
||||
]
|
||||
|
||||
detailed_known_urls_metadata.append( detailed_known_urls_metadata_row )
|
||||
|
|
|
|||
|
|
@ -388,7 +388,7 @@ class TestClientDB( unittest.TestCase ):
|
|||
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_ARCHIVE, None, 0 ) )
|
||||
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_DURATION, ClientSearch.NumberTest.STATICCreateFromCharacters( '<', 100, ), 0 ) )
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_DURATION, ClientSearch.NumberTest.STATICCreateFromCharacters( '<', 100, ), 1 ) )
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_DURATION, ClientSearch.NumberTest.STATICCreateFromCharacters( '<', 0, ), 0 ) )
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_DURATION, ClientSearch.NumberTest.STATICCreateFromCharacters( HC.UNICODE_APPROX_EQUAL, 100, ), 0 ) )
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_DURATION, ClientSearch.NumberTest.STATICCreateFromCharacters( HC.UNICODE_APPROX_EQUAL, 0, ), 1 ) )
|
||||
|
|
@ -472,7 +472,7 @@ class TestClientDB( unittest.TestCase ):
|
|||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_NUM_TAGS, ( '', '>', 0 ), 0 ) )
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_NUM_TAGS, ( '', '>', 1 ), 0 ) )
|
||||
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_NUM_WORDS, ClientSearch.NumberTest.STATICCreateFromCharacters( '<', 1 ), 0 ) )
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_NUM_WORDS, ClientSearch.NumberTest.STATICCreateFromCharacters( '<', 1 ), 1 ) )
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_NUM_WORDS, ClientSearch.NumberTest.STATICCreateFromCharacters( '<', 0 ), 0 ) )
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_NUM_WORDS, ClientSearch.NumberTest.STATICCreateFromCharacters( HC.UNICODE_APPROX_EQUAL, 0 ), 1 ) )
|
||||
tests.append( ( ClientSearch.PREDICATE_TYPE_SYSTEM_NUM_WORDS, ClientSearch.NumberTest.STATICCreateFromCharacters( HC.UNICODE_APPROX_EQUAL, 1 ), 0 ) )
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ from hydrus.client.networking import ClientNetworking
|
|||
from hydrus.client.networking import ClientNetworkingBandwidth
|
||||
from hydrus.client.networking import ClientNetworkingContexts
|
||||
from hydrus.client.networking import ClientNetworkingDomain
|
||||
from hydrus.client.networking import ClientNetworkingFunctions
|
||||
from hydrus.client.networking import ClientNetworkingJobs
|
||||
from hydrus.client.networking import ClientNetworkingLogin
|
||||
from hydrus.client.networking import ClientNetworkingSessions
|
||||
|
|
@ -281,51 +282,20 @@ class TestURLClasses( unittest.TestCase ):
|
|||
|
||||
def test_encoding( self ):
|
||||
|
||||
name = 'test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
preferred_scheme = 'https'
|
||||
netloc = 'testbooru.cx'
|
||||
human_url = 'https://testbooru.cx/post/page.php?id=1234 56&s=view'
|
||||
encoded_url = 'https://testbooru.cx/post/page.php?id=1234%2056&s=view'
|
||||
|
||||
alphabetise_get_parameters = True
|
||||
match_subdomains = False
|
||||
keep_matched_subdomains = False
|
||||
can_produce_multiple_files = False
|
||||
should_be_associated_with_files = True
|
||||
keep_fragment = False
|
||||
self.assertEqual( ClientNetworkingFunctions.WashURL( human_url ), encoded_url )
|
||||
self.assertEqual( ClientNetworkingFunctions.WashURL( encoded_url ), encoded_url )
|
||||
|
||||
path_components = []
|
||||
human_url_with_fragment = 'https://testbooru.cx/post/page.php?id=1234 56&s=view#hello'
|
||||
encoded_url_with_fragment = 'https://testbooru.cx/post/page.php?id=1234%2056&s=view#hello'
|
||||
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
|
||||
self.assertEqual( ClientNetworkingFunctions.WashURL( human_url_with_fragment ), encoded_url_with_fragment )
|
||||
self.assertEqual( ClientNetworkingFunctions.WashURL( encoded_url_with_fragment ), encoded_url_with_fragment )
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ) ) )
|
||||
|
||||
send_referral_url = ClientNetworkingURLClass.SEND_REFERRAL_URL_ONLY_IF_PROVIDED
|
||||
referral_url_converter = None
|
||||
gallery_index_type = None
|
||||
gallery_index_identifier = None
|
||||
gallery_index_delta = 1
|
||||
example_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
# encoding test
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_ANY, example_string = 'hello' ) ) )
|
||||
|
||||
url_class = ClientNetworkingURLClass.URLClass( name, url_type = url_type, preferred_scheme = preferred_scheme, netloc = netloc, path_components = path_components, parameters = parameters, send_referral_url = send_referral_url, referral_url_converter = referral_url_converter, gallery_index_type = gallery_index_type, gallery_index_identifier = gallery_index_identifier, gallery_index_delta = gallery_index_delta, example_url = example_url )
|
||||
|
||||
url_class.SetURLBooleans( match_subdomains, keep_matched_subdomains, alphabetise_get_parameters, can_produce_multiple_files, should_be_associated_with_files, keep_fragment )
|
||||
|
||||
unnormalised_human_url = 'https://testbooru.cx/post/page.php?id=1234 56&s=view'
|
||||
normalised_encoded_url = 'https://testbooru.cx/post/page.php?id=1234%2056&s=view'
|
||||
|
||||
self.assertEqual( url_class.Normalise( unnormalised_human_url ), normalised_encoded_url )
|
||||
self.assertEqual( url_class.Normalise( normalised_encoded_url ), normalised_encoded_url )
|
||||
self.assertEqual( ClientNetworkingFunctions.WashURL( human_url_with_fragment, keep_fragment = False ), encoded_url )
|
||||
self.assertEqual( ClientNetworkingFunctions.WashURL( encoded_url_with_fragment, keep_fragment = False ), encoded_url )
|
||||
|
||||
|
||||
def test_defaults( self ):
|
||||
|
|
|
|||
|
|
@ -15,10 +15,17 @@ if ! source venv/bin/activate; then
|
|||
fi
|
||||
|
||||
# You can copy this file to 'client-user.sh' and add in your own launch parameters here if you like, and a git pull won't overwrite the file.
|
||||
# Just tack new params on like this:
|
||||
# python hydrus_client.py -d="/path/to/hydrus/db"
|
||||
# Just tack new hardcoded params on like this:
|
||||
#
|
||||
# python hydrus_client.py -d="/path/to/hydrus/db" "$@"
|
||||
#
|
||||
# The "$@" part also passes on any launch parameters this script was called with, so you can also just go--
|
||||
#
|
||||
# ./hydrus_client.sh -d="/path/to/hydrus/db"
|
||||
#
|
||||
# --depending on your needs!
|
||||
|
||||
python hydrus_client.py
|
||||
python hydrus_client.py "$@"
|
||||
|
||||
deactivate
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,26 @@
|
|||
|
||||
pushd "%~dp0"
|
||||
|
||||
IF "%1"=="" (
|
||||
|
||||
set python_bin=python
|
||||
|
||||
) else (
|
||||
|
||||
set python_bin=%1
|
||||
|
||||
)
|
||||
|
||||
IF "%2"=="" (
|
||||
|
||||
set venv_location=venv
|
||||
|
||||
) else (
|
||||
|
||||
set venv_location=%2
|
||||
|
||||
)
|
||||
|
||||
ECHO r::::::::::::::::::::::::::::::::::r
|
||||
ECHO : :
|
||||
ECHO : :PP. :
|
||||
|
|
@ -27,24 +47,27 @@ ECHO:
|
|||
ECHO hydrus
|
||||
ECHO:
|
||||
|
||||
where /q python
|
||||
IF ERRORLEVEL 1 (
|
||||
IF "%python_bin%"=="python" (
|
||||
|
||||
SET /P gumpf="You do not seem to have python installed. Please check the 'running from source' help."
|
||||
where /q %python_bin%
|
||||
IF ERRORLEVEL 1 (
|
||||
|
||||
popd
|
||||
SET /P gumpf="You do not seem to have python installed. Please check the 'running from source' help."
|
||||
|
||||
EXIT /B 1
|
||||
popd
|
||||
|
||||
EXIT /B 1
|
||||
|
||||
)
|
||||
)
|
||||
|
||||
IF EXIST "venv\" (
|
||||
IF EXIST "%venv_location%\" (
|
||||
|
||||
SET /P ready="Virtual environment will be reinstalled. Hit Enter to start."
|
||||
|
||||
echo Deleting old venv...
|
||||
|
||||
rmdir /s /q venv
|
||||
rmdir /s /q %venv_location%
|
||||
|
||||
) ELSE (
|
||||
|
||||
|
|
@ -52,7 +75,7 @@ IF EXIST "venv\" (
|
|||
|
||||
)
|
||||
|
||||
IF EXIST "venv\" (
|
||||
IF EXIST "%venv_location%\" (
|
||||
|
||||
SET /P gumpf="It looks like the venv directory did not delete correctly. Do you have it activated in a terminal or IDE anywhere? Please close that and try this again!"
|
||||
|
||||
|
|
@ -68,7 +91,7 @@ ECHO --------
|
|||
ECHO Users on older Windows or Python ^>=3.11 need the advanced install.
|
||||
ECHO:
|
||||
ECHO Your Python version is:
|
||||
python --version
|
||||
%python_bin% --version
|
||||
ECHO:
|
||||
SET /P install_type="Do you want the (s)imple or (a)dvanced install? "
|
||||
|
||||
|
|
@ -161,9 +184,9 @@ goto :parse_fail
|
|||
ECHO --------
|
||||
echo Creating new venv...
|
||||
|
||||
python -m venv venv
|
||||
%python_bin% -m venv %venv_location%
|
||||
|
||||
CALL venv\Scripts\activate.bat
|
||||
CALL %venv_location%\Scripts\activate.bat
|
||||
|
||||
IF ERRORLEVEL 1 (
|
||||
|
||||
|
|
@ -252,7 +275,7 @@ IF "%install_type%" == "a" (
|
|||
|
||||
)
|
||||
|
||||
CALL venv\Scripts\deactivate.bat
|
||||
CALL %venv_location%\Scripts\deactivate.bat
|
||||
|
||||
ECHO --------
|
||||
SET /P done="Done!"
|
||||
|
|
|
|||
Loading…
Reference in New Issue