Merge branch 'develop'
This commit is contained in:
Hydrus Network Developer
commit
0f6c01bb06
|
|
@ -7,7 +7,7 @@ title: Changelog
|
|||
!!! note
|
||||
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
|
||||
|
||||
## [Version 567](https://github.com/hydrusnetwork/hydrus/releases/tag/v567)
|
||||
## [Version 568](https://github.com/hydrusnetwork/hydrus/releases/tag/v568)
|
||||
|
||||
### user contributions
|
||||
|
||||
|
|
@ -16,35 +16,73 @@ title: Changelog
|
|||
* thanks to a user, setting the Qt style in *options->style* should be more reliable (fixing some name case sensitivity issues)
|
||||
* thanks to a user, there's a new 'default' dark mode QSS stylesheet that has nicer valid/invalid colours. we'll build on this and try to detect dark mode better in future and auto-switch to this as the base when the application is in dark mode.
|
||||
|
||||
### misc
|
||||
### misc improvements
|
||||
|
||||
* added a 'tag in reverse' checkbox to the new incremental tagger panel. this simply applies the given iterator to the last file first and then works backwards, e.g. 5, 4, 3, 2, 1 for start=1, step=1 on five files
|
||||
* all _new_ system:url predicates will have slightly different (standardised) labels, and all these labels should parse correctly in the system predicate parser if you copy/paste
|
||||
* you should now be able to enter 'system:has url matching regex (regex with upper case)' and 'system:has url (url with upper case)' and it'll propage through parsing. this definitely has not™ broken any other predicate parsing. you can enter url class names with upper case if you want, but url class names should now match regardless of letter case
|
||||
* you can now open the 'extra info' button (up top of a media viewer) on a jpeg if that jpeg has no exif or other human-readable metadata (to see just the progressive and subsampling info)
|
||||
* the file log's right-click menu, the part where it says 'additional urls', is now more compact and will show the 'request url', if that differs from the main url, either because of the new ephemeral parameters or an api/redirect. it is now much easier to debug the various 'what was actually sent to the server?' problems!
|
||||
* you should now be able to enter 'system:has url matching regex (regex with upper case)' and 'system:has url (url with upper case)' and it'll propagate through parsing. this definitely has not™ broken any other predicate parsing. you can enter url class names with upper case if you want, but url class names should now match regardless of letter case
|
||||
* if you have added, edited, or deleted any url classes and try to cancel the 'manage url classes' dialog, it will now ask if that is correct
|
||||
* added a new EXPERIMENTAL checkbox to _options->tag presentation_ that will replace emojis and other unicode symbol garbage with □. if you have crazy rendering for emoji stuff, try it out
|
||||
* the tag summary generators that make thumbnail banners now wash their tags through the 'render tag for user' system, which will apply this new emoji rule and 'replace underscores with spaces'
|
||||
* added the 'rating' parser from the default gelbooru 0.2.5 parser to the 0.2.0 parser; this should add for more 'rating' parsing from a variety of boorus
|
||||
|
||||
### misc fixes
|
||||
|
||||
* fixed a typo bug when deleting domain-based timestamps in the edit times dialog
|
||||
* fixed the 'system:has url matching class (blah)' predicate edit panel's initialisation. it was always initialising to the top of the list, not remembering the 'default' or 'I want to edit this' value it was initialising with
|
||||
* 'manage urls' now asks if it is ok to ok if you have any text still in the input
|
||||
* you can now open the 'extra info' button (up top of a media viewer) on a jpeg if that jpeg has no exif or other human-readable metadata (to see just the progressive and subsampling info)
|
||||
* updated the QuickSync link to its new home at https://breadthread.duckdns.org/
|
||||
|
||||
### append random text
|
||||
|
||||
* the String Converter has a new step type: 'append random text'. you supply the population (e.g. '0123456789abcdef') and the number of characters (e.g. 16), and it will append 'b2f96e8eda457a1e', and then the next time you check, '1fa591ad9786ea3b', etc... useful if you want to, say, make up a new token
|
||||
|
||||
### URL storage/display changes
|
||||
|
||||
* today I correct a foolish decision I made when I first implemented the hydrus downloader engine--handling and storing URLs internally as 'pretty' decoded text, rather than with the proper ugly '%20" stuff you sometimes see. this improves support for weird URLs and makes some behind the scenes things simpler. you do not need to make any changes, but there is a chance some particularly funky URLs will redownload once more if your subscription runs into them again (this change breaks some 'known url' checking logic, since what is stored is now slightly different, but this 99% doesn't affect Post URLs, so no big worries)
|
||||
* so, URLs are no longer decoded in the normalisation step. they are now saved in the file log as their proper actual 'what is sent to the server' encoded text. it will display in UI as the pretty version, but if you copy to clipboard, you get the data version--pretty much how your web browser address bar works. I have made it show 'pretty' in the file log and search log lists, 'copy url' menu labels, and hyperlink tooltips, but in the more technical 'manage GUGs' and so on, it shows the data version. let me know if I have forgotten to display them pretty anywhere!
|
||||
* when you paste a URL, some new normalisation tech tries to figure out if it is pre-encoded or not
|
||||
* there's also some GUG work. when you enter a query text like `male/female` or `blonde_hair%20blue_eyes`, some new logic tries to infer whether what you entered is encoded or not. it should handle pretty much everything well unless you have a single-tag query with a legit percent character in the middle (in which case you'll have to enter `%25` instead, but we'll see if it ever happens)
|
||||
* today I correct a foolish decision I made when I first implemented the hydrus downloader engine--handling and storing URLs internally as 'pretty' decoded text, rather than with the proper ugly '%20" stuff you sometimes see. I now store urls as the 'encoded' variant all the time, and only convert to the pretty version when the user sees it. this improves support for weird URLs and simplifies some behind the scenes. you do not need to do anything, and everything should work pretty much as before, but there is a chance some particularly funky URLs will redownload one more time if your subscription runs into them again (this change breaks some 'known url' checking logic, since what is stored is now slightly different, but this 99% doesn't affect Post URLs, so no big worries)
|
||||
* so, while URLs still show pretty in a file/search log, if you copy them to clipboard, you now get the encoded version--pretty much how your web browser address bar works. I have made it show 'pretty' in the file log and search log lists, 'copy url' menu labels, and hyperlink tooltips, but in the more technical 'manage url classes' and 'manage GUGs' and so on where you are actually editing a URL, it shows the encoded version. let me know if I have forgotten to display them pretty anywhere!
|
||||
* **IF YOU ARE AN ADVANCED USER WHO MAKES CRAZY URL CLASSES:** since URLs are now stored as the %-encoded version in all cases, component and parameter tests now apply to %-encoding (e.g. you are now testing for `post%5Bid%5D`, not `post[id]`). when your URL Classes update this week, I convert existing path component defaults, parameter names and defaults, and `fixed_text` String Matches for path component names and parameter values to their %-encoded value. I hope this will provide for a clean transition where it matters. unfortunately, if the String Matches were a regex or you were pulling a rabbit out of your hat with edge-case pre-%-encoded default values, I just can't auto-convert that, so please scroll down your crazier URL Classes and see if any say they don't match their example URLs!
|
||||
* there's also some GUG work. when you enter a query text like `male/female` or `blonde_hair%20blue_eyes`, some new logic tries to infer whether what you entered is pre-encoded or not. it should handle pretty much everything well unless you have a single-tag query with a legit percent character in the middle (in which case you'll have to enter `%25` instead, but we'll see if it ever happens)
|
||||
* these changes simplify the url parsing routine, eliminating plenty of nonsense hackery I've inserted over the years to make things like `6+girls blonde_hair`/`6%2Bgirls+blonde_hair` work with a merged system. this has mostly been a delicate cleanup job; long planned, finally triggered
|
||||
|
||||
### ephemeral URL parameters
|
||||
### allow all ephemeral parameters
|
||||
|
||||
* I was going to roll out 'ephemeral token' parameters, and I basically had it done, but I realised late that I was being stupid in a brand new way, basically expanding the whitelist when turning off the blacklist was a nicer solution. I will work on this more next week, I think ultimately making it so Post URLs are not clipped of undefined parameters before they are is sent to the server, just like for Gallery URLs. I will separately introduce 'I just need to add some random hex in this parameter to tell this cache I want the original' under different tech
|
||||
* so, I did some behind the scenes URL filtering tech, and file import objects handle full and stripped down versions of Post URLs, but it doesn't do much yet
|
||||
* URL Classes have a new checkbox, 'keep extra parameters for server', which will determine whether URLs should hang on to undefined parameters in the first stage of normalisation, which governs what is sent to the server. this is now default True on all new URL Classes! existing URL Classes will default True only if the URL Class is a Gallery/Watchable URL without an API/redirect converter (which was essentially the previous hardcoded behaviour). you cannot set this value if the URL has an API/redirect converter
|
||||
|
||||
### boring cleanup
|
||||
### allow specific ephemeral parameters
|
||||
|
||||
* I cleaned up some URL Class code
|
||||
* the URL Class has a new buddy 'Parameter' class to handle param testing
|
||||
* alternately, you can now specify single 'ephemeral token' parameters in the new parameter edit dialog. it is just a check box that says 'use this for the request, but don't save it'. these _are_ kept for the API/redirect URL
|
||||
* if you are feeling extremely big brain, there is now a String Processor for the default value, if both 'is ephemeral?' is checked and 'default value' is not 'None'. this lets you append/replace your fixed default value with the current time, or, now, just some random hex or something! hence we can now define our own basic one-time token generators for telling caches to give us original quality etc...
|
||||
|
||||
### manage url classes dialog
|
||||
|
||||
* there's a new read-only text field with the 'example url' and 'normalised url' section called 'request url'. this shows either the example URL with its extra, ephemeral parameters, or it will show the API/redirect URL. it shows what will be sent to the server
|
||||
* URL Class parameters now have their own edit panel, with everything available in one place, rather than the three-dialogs-in-a-row mess of before. also, the name and value widgets have locked normal/%-encoded text inputs that will live update each other, so you can paste whatever is convenient for you and see a preview either way
|
||||
* URL Class path components also have their own edit panel. same deal as for parameters, but a little simpler
|
||||
|
||||
### client api
|
||||
|
||||
* the `/add_urls/get_url_info` command now returns `request_url` value, which is either the 'for server' normalised URL, which may include ephemeral tokens, or the API/redirect URL, just as in the new 'manage url classes' dialog
|
||||
* the `/add_files/undelete_files` command now filters the files you give it to make sure that they are actually in your file storage. no more undeleting files you don't have!
|
||||
* added a new `/add_files/clear_file_deletion_record` command, which erases deletion records for physically deleted files
|
||||
* updated api help docs and unit tests for the above
|
||||
* client api version is now 63
|
||||
|
||||
### boring stuff
|
||||
|
||||
* the client is now much more robust if any of its URL Classes do not match their own example URLs. it will boot, to start with (lol), and you can now open the 'manage url classes' dialog without UI error popups. manage url classes now notes which URL Classes do not match their own example URLs, for easy skimming
|
||||
* the 'URL Class' class has a new buddy 'Parameter' class to handle param testing
|
||||
* simplified some of the guts of URL normalisation, from path/param clipping to how API URL generation is navigated
|
||||
* rewrote how the query string of a URL is deconstructed and scanned against your parameters. less chance of edge-case errors/merges and easier to expand in future
|
||||
* brushed up the URL Class unit tests to account for the above changes and added new tests for encoding, ephemeral, and default parameter values (which must have been missed a long time ago)
|
||||
* when you paste a URL, some new normalisation tech tries to figure out if it is pre-encoded or not
|
||||
* brushed up the URL Class unit tests to account for the above changes and added new tests for encoding, 'is ephemeral', 'keep extra params for server', default parameter string processors, and simple default parameter values (which must have been missed a long time ago)
|
||||
* also broke the monolithic url class unit test into eight smaller (albeit ugly for now) pieces
|
||||
* added a unit test for the new 'append random text' converter
|
||||
* cleaned up some misc URL Class code
|
||||
|
||||
## Version 567 was cancelled, its changes folded into 568.
|
||||
|
||||
## [Version 566](https://github.com/hydrusnetwork/hydrus/releases/tag/v566)
|
||||
|
||||
|
|
|
|||
|
|
@ -555,7 +555,7 @@ If you specify a file service, the file will only be deleted from that location.
|
|||
|
||||
### **POST `/add_files/undelete_files`** { id="add_files_undelete_files" }
|
||||
|
||||
_Tell the client to pull files back out of the trash._
|
||||
_Tell the client to restore files that were previously deleted to their old file service(s)._
|
||||
|
||||
Restricted access:
|
||||
: YES. Import Files permission needed.
|
||||
|
|
@ -577,11 +577,37 @@ Arguments (in JSON):
|
|||
|
||||
Response:
|
||||
: 200 and no content.
|
||||
|
||||
This is the reverse of a delete_files--restoring files back to where they came from. If you specify a file service, the files will only be undeleted to there (if they have a delete record, otherwise this is nullipotent). The default, 'all my files', undeletes to all local file services for which there are deletion records.
|
||||
|
||||
This operation will only occur on files that are currently in your file store (i.e. in 'all local files', and maybe, but not necessarily, in 'trash'). You cannot 'undelete' something you do not have!
|
||||
|
||||
### **POST `/add_files/clear_file_deletion_record`** { id="add_files_clear_file_deletion_record" }
|
||||
|
||||
_Tell the client to forget that it once deleted files._
|
||||
|
||||
Restricted access:
|
||||
: YES. Import Files permission needed.
|
||||
|
||||
You can use hash or hashes, whichever is more convenient.
|
||||
|
||||
This is the reverse of a delete_files--removing files from trash and putting them back where they came from. If you specify a file service, the files will only be undeleted to there (if they have a delete record, otherwise this is nullipotent). The default, 'all my files', undeletes to all local file services for which there are deletion records. There is no error if any of the files do not currently exist in 'trash'.
|
||||
|
||||
Required Headers:
|
||||
:
|
||||
* `Content-Type`: application/json
|
||||
|
||||
Arguments (in JSON):
|
||||
:
|
||||
* [files](#parameters_files)
|
||||
|
||||
```json title="Example request body"
|
||||
{
|
||||
"hash" : "78f92ba4a786225ee2a1236efa6b7dc81dd729faf4af99f96f3e20bad6d8b538"
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
: 200 and no content.
|
||||
|
||||
This is the same as the advanced deletion option of the same basic name. It will erase the record that a file has been physically deleted (i.e. it only applies to deletion records in the 'all local files' domain). A file that no longer has a 'all local files' deletion record will pass a 'exclude previously deleted files' check in a _file import options_.
|
||||
|
||||
|
||||
### **POST `/add_files/archive_files`** { id="add_files_archive_files" }
|
||||
|
||||
|
|
@ -740,16 +766,17 @@ Arguments:
|
|||
* `url`: (the url you want to ask about)
|
||||
|
||||
Example request:
|
||||
: for URL `https://8ch.net/tv/res/1846574.html`:
|
||||
: for URL `https://boards.4chan.org/tv/thread/197641945/itt-moments-in-film-or-tv-that-aged-poorly`:
|
||||
```
|
||||
/add_urls/get_url_info?url=https%3A%2F%2F8ch.net%2Ftv%2Fres%2F1846574.html
|
||||
/add_urls/get_url_info?url=https%3A%2F%2Fboards.4chan.org%2Ftv%2Fthread%2F197641945%2Fitt-moments-in-film-or-tv-that-aged-poorly
|
||||
```
|
||||
|
||||
Response:
|
||||
: Some JSON describing what the client thinks of the URL.
|
||||
```json title="Example response"
|
||||
{
|
||||
"normalised_url" : "https://8ch.net/tv/res/1846574.html",
|
||||
"request_url" : "https://a.4cdn.org/tv/thread/197641945.json",
|
||||
"normalised_url" : "https://boards.4chan.org/tv/thread/197641945",
|
||||
"url_type" : 4,
|
||||
"url_type_string" : "watchable url",
|
||||
"match_name" : "8chan thread",
|
||||
|
|
@ -767,6 +794,10 @@ Response:
|
|||
|
||||
'Unknown' URLs are treated in the client as direct File URLs. Even though the 'File URL' type is available, most file urls do not have a URL Class, so they will appear as Unknown. Adding them to the client will pass them to the URL Downloader as a raw file for download and import.
|
||||
|
||||
The `normalised_url` is the fully normalised URL--what is used for comparison and saving to disk.
|
||||
|
||||
The `request_url` is either the lighter 'for server' normalised URL, which may include ephemeral token parameters, or, as in the case here, the fully converted API/redirect URL. (When hydrus is asked to check a 4chan thread, it doesn't hit the HTML, but the JSON API.)
|
||||
|
||||
|
||||
### **POST `/add_urls/add_url`** { id="add_urls_add_url" }
|
||||
|
||||
|
|
|
|||
|
|
@ -35,35 +35,62 @@
|
|||
<h1 id="changelog"><a href="#changelog">changelog</a></h1>
|
||||
<ul>
|
||||
<li>
|
||||
<h2 id="version_567"><a href="#version_567">version 567</a></h2>
|
||||
<h2 id="version_568"><a href="#version_568">version 568</a></h2>
|
||||
<p><i>Version 567 was cancelled, its changes folded into 568</i></p>
|
||||
<ul>
|
||||
<li><h3>user contributions</h3></li>
|
||||
<li>thanks to a user, the new docx, pptx, and xlsx support is improved, with better thumbnails (better ratio, better icon itself, and sometimes an actual preview thumbnail for pptx), better file detection (fewer false positives with stuff like ppt templates), and word count for docx and pptx. I am queueing everyone's existing docx and pptx files for a metadata rescan and thumbnail regen on update</li>
|
||||
<li>thanks to a user, the cbz scanner now ignores the `__MACOSX` folder</li>
|
||||
<li>thanks to a user, setting the Qt style in *options->style* should be more reliable (fixing some name case sensitivity issues)</li>
|
||||
<li>thanks to a user, there's a new 'default' dark mode QSS stylesheet that has nicer valid/invalid colours. we'll build on this and try to detect dark mode better in future and auto-switch to this as the base when the application is in dark mode.</li>
|
||||
<li><h3>misc</h3></li>
|
||||
<li><h3>misc improvements</h3></li>
|
||||
<li>added a 'tag in reverse' checkbox to the new incremental tagger panel. this simply applies the given iterator to the last file first and then works backwards, e.g. 5, 4, 3, 2, 1 for start=1, step=1 on five files</li>
|
||||
<li>all _new_ system:url predicates will have slightly different (standardised) labels, and all these labels should parse correctly in the system predicate parser if you copy/paste</li>
|
||||
<li>you should now be able to enter 'system:has url matching regex (regex with upper case)' and 'system:has url (url with upper case)' and it'll propage through parsing. this definitely has not™ broken any other predicate parsing. you can enter url class names with upper case if you want, but url class names should now match regardless of letter case</li>
|
||||
<li>you can now open the 'extra info' button (up top of a media viewer) on a jpeg if that jpeg has no exif or other human-readable metadata (to see just the progressive and subsampling info)</li>
|
||||
<li>the file log's right-click menu, the part where it says 'additional urls', is now more compact and will show the 'request url', if that differs from the main url, either because of the new ephemeral parameters or an api/redirect. it is now much easier to debug the various 'what was actually sent to the server?' problems!</li>
|
||||
<li>you should now be able to enter 'system:has url matching regex (regex with upper case)' and 'system:has url (url with upper case)' and it'll propagate through parsing. this definitely has not™ broken any other predicate parsing. you can enter url class names with upper case if you want, but url class names should now match regardless of letter case</li>
|
||||
<li>if you have added, edited, or deleted any url classes and try to cancel the 'manage url classes' dialog, it will now ask if that is correct</li>
|
||||
<li>added a new EXPERIMENTAL checkbox to _options->tag presentation_ that will replace emojis and other unicode symbol garbage with □. if you have crazy rendering for emoji stuff, try it out</li>
|
||||
<li>the tag summary generators that make thumbnail banners now wash their tags through the 'render tag for user' system, which will apply this new emoji rule and 'replace underscores with spaces'</li>
|
||||
<li>added the 'rating' parser from the default gelbooru 0.2.5 parser to the 0.2.0 parser; this should add for more 'rating' parsing from a variety of boorus</li>
|
||||
<li><h3>misc fixes</h3></li>
|
||||
<li>fixed a typo bug when deleting domain-based timestamps in the edit times dialog</li>
|
||||
<li>fixed the 'system:has url matching class (blah)' predicate edit panel's initialisation. it was always initialising to the top of the list, not remembering the 'default' or 'I want to edit this' value it was initialising with</li>
|
||||
<li>'manage urls' now asks if it is ok to ok if you have any text still in the input</li>
|
||||
<li>you can now open the 'extra info' button (up top of a media viewer) on a jpeg if that jpeg has no exif or other human-readable metadata (to see just the progressive and subsampling info)</li>
|
||||
<li>updated the QuickSync link to its new home at https://breadthread.duckdns.org/</li>
|
||||
<li><h3>append random text</h3></li>
|
||||
<li>the String Converter has a new step type: 'append random text'. you supply the population (e.g. '0123456789abcdef') and the number of characters (e.g. 16), and it will append 'b2f96e8eda457a1e', and then the next time you check, '1fa591ad9786ea3b', etc... useful if you want to, say, make up a new token</li>
|
||||
<li><h3>URL storage/display changes</h3></li>
|
||||
<li>today I correct a foolish decision I made when I first implemented the hydrus downloader engine--handling and storing URLs internally as 'pretty' decoded text, rather than with the proper ugly '%20" stuff you sometimes see. this improves support for weird URLs and makes some behind the scenes things simpler. you do not need to make any changes, but there is a chance some particularly funky URLs will redownload once more if your subscription runs into them again (this change breaks some 'known url' checking logic, since what is stored is now slightly different, but this 99% doesn't affect Post URLs, so no big worries)</li>
|
||||
<li>so, URLs are no longer decoded in the normalisation step. they are now saved in the file log as their proper actual 'what is sent to the server' encoded text. it will display in UI as the pretty version, but if you copy to clipboard, you get the data version--pretty much how your web browser address bar works. I have made it show 'pretty' in the file log and search log lists, 'copy url' menu labels, and hyperlink tooltips, but in the more technical 'manage GUGs' and so on, it shows the data version. let me know if I have forgotten to display them pretty anywhere!</li>
|
||||
<li>when you paste a URL, some new normalisation tech tries to figure out if it is pre-encoded or not</li>
|
||||
<li>there's also some GUG work. when you enter a query text like `male/female` or `blonde_hair%20blue_eyes`, some new logic tries to infer whether what you entered is encoded or not. it should handle pretty much everything well unless you have a single-tag query with a legit percent character in the middle (in which case you'll have to enter `%25` instead, but we'll see if it ever happens)</li>
|
||||
<li>today I correct a foolish decision I made when I first implemented the hydrus downloader engine--handling and storing URLs internally as 'pretty' decoded text, rather than with the proper ugly '%20" stuff you sometimes see. I now store urls as the 'encoded' variant all the time, and only convert to the pretty version when the user sees it. this improves support for weird URLs and simplifies some behind the scenes. you do not need to do anything, and everything should work pretty much as before, but there is a chance some particularly funky URLs will redownload one more time if your subscription runs into them again (this change breaks some 'known url' checking logic, since what is stored is now slightly different, but this 99% doesn't affect Post URLs, so no big worries)</li>
|
||||
<li>so, while URLs still show pretty in a file/search log, if you copy them to clipboard, you now get the encoded version--pretty much how your web browser address bar works. I have made it show 'pretty' in the file log and search log lists, 'copy url' menu labels, and hyperlink tooltips, but in the more technical 'manage url classes' and 'manage GUGs' and so on where you are actually editing a URL, it shows the encoded version. let me know if I have forgotten to display them pretty anywhere!</li>
|
||||
<li>**IF YOU ARE AN ADVANCED USER WHO MAKES CRAZY URL CLASSES:** since URLs are now stored as the %-encoded version in all cases, component and parameter tests now apply to %-encoding (e.g. you are now testing for `post%5Bid%5D`, not `post[id]`). when your URL Classes update this week, I convert existing path component defaults, parameter names and defaults, and `fixed_text` String Matches for path component names and parameter values to their %-encoded value. I hope this will provide for a clean transition where it matters. unfortunately, if the String Matches were a regex or you were pulling a rabbit out of your hat with edge-case pre-%-encoded default values, I just can't auto-convert that, so please scroll down your crazier URL Classes and see if any say they don't match their example URLs!</li>
|
||||
<li>there's also some GUG work. when you enter a query text like `male/female` or `blonde_hair%20blue_eyes`, some new logic tries to infer whether what you entered is pre-encoded or not. it should handle pretty much everything well unless you have a single-tag query with a legit percent character in the middle (in which case you'll have to enter `%25` instead, but we'll see if it ever happens)</li>
|
||||
<li>these changes simplify the url parsing routine, eliminating plenty of nonsense hackery I've inserted over the years to make things like `6+girls blonde_hair`/`6%2Bgirls+blonde_hair` work with a merged system. this has mostly been a delicate cleanup job; long planned, finally triggered</li>
|
||||
<li><h3>ephemeral URL parameters</h3></li>
|
||||
<li>I was going to roll out 'ephemeral token' parameters, and I basically had it done, but I realised late that I was being stupid in a brand new way, basically expanding the whitelist when turning off the blacklist was a nicer solution. I will work on this more next week, I think ultimately making it so Post URLs are not clipped of undefined parameters before they are is sent to the server, just like for Gallery URLs. I will separately introduce 'I just need to add some random hex in this parameter to tell this cache I want the original' under different tech</li>
|
||||
<li>so, I did some behind the scenes URL filtering tech, and file import objects handle full and stripped down versions of Post URLs, but it doesn't do much yet</li>
|
||||
<li><h3>boring cleanup</h3></li>
|
||||
<li>I cleaned up some URL Class code</li>
|
||||
<li>the URL Class has a new buddy 'Parameter' class to handle param testing</li>
|
||||
<li><h3>allow all ephemeral parameters</h3></li>
|
||||
<li>URL Classes have a new checkbox, 'keep extra parameters for server', which will determine whether URLs should hang on to undefined parameters in the first stage of normalisation, which governs what is sent to the server. this is now default True on all new URL Classes! existing URL Classes will default True only if the URL Class is a Gallery/Watchable URL without an API/redirect converter (which was essentially the previous hardcoded behaviour). you cannot set this value if the URL has an API/redirect converter</li>
|
||||
<li><h3>allow specific ephemeral parameters</h3></li>
|
||||
<li>alternately, you can now specify single 'ephemeral token' parameters in the new parameter edit dialog. it is just a check box that says 'use this for the request, but don't save it'. these _are_ kept for the API/redirect URL</li>
|
||||
<li>if you are feeling extremely big brain, there is now a String Processor for the default value, if both 'is ephemeral?' is checked and 'default value' is not 'None'. this lets you append/replace your fixed default value with the current time, or, now, just some random hex or something! hence we can now define our own basic one-time token generators for telling caches to give us original quality etc...</li>
|
||||
<li><h3>manage url classes dialog</h3></li>
|
||||
<li>there's a new read-only text field with the 'example url' and 'normalised url' section called 'request url'. this shows either the example URL with its extra, ephemeral parameters, or it will show the API/redirect URL. it shows what will be sent to the server</li>
|
||||
<li>URL Class parameters now have their own edit panel, with everything available in one place, rather than the three-dialogs-in-a-row mess of before. also, the name and value widgets have locked normal/%-encoded text inputs that will live update each other, so you can paste whatever is convenient for you and see a preview either way</li>
|
||||
<li>URL Class path components also have their own edit panel. same deal as for parameters, but a little simpler</li>
|
||||
<li><h3>client api</h3></li>
|
||||
<li>the `/add_urls/get_url_info` command now returns `request_url` value, which is either the 'for server' normalised URL, which may include ephemeral tokens, or the API/redirect URL, just as in the new 'manage url classes' dialog</li>
|
||||
<li>the `/add_files/undelete_files` command now filters the files you give it to make sure that they are actually in your file storage. no more undeleting files you don't have!</li>
|
||||
<li>added a new `/add_files/clear_file_deletion_record` command, which erases deletion records for physically deleted files</li>
|
||||
<li>updated api help docs and unit tests for the above</li>
|
||||
<li>client api version is now 63</li>
|
||||
<li><h3>boring stuff</h3></li>
|
||||
<li>the client is now much more robust if any of its URL Classes do not match their own example URLs. it will boot, to start with (lol), and you can now open the 'manage url classes' dialog without UI error popups. manage url classes now notes which URL Classes do not match their own example URLs, for easy skimming</li>
|
||||
<li>the 'URL Class' class has a new buddy 'Parameter' class to handle param testing</li>
|
||||
<li>simplified some of the guts of URL normalisation, from path/param clipping to how API URL generation is navigated</li>
|
||||
<li>rewrote how the query string of a URL is deconstructed and scanned against your parameters. less chance of edge-case errors/merges and easier to expand in future</li>
|
||||
<li>brushed up the URL Class unit tests to account for the above changes and added new tests for encoding, ephemeral, and default parameter values (which must have been missed a long time ago)</li>
|
||||
<li>when you paste a URL, some new normalisation tech tries to figure out if it is pre-encoded or not</li>
|
||||
<li>brushed up the URL Class unit tests to account for the above changes and added new tests for encoding, 'is ephemeral', 'keep extra params for server', default parameter string processors, and simple default parameter values (which must have been missed a long time ago)</li>
|
||||
<li>also broke the monolithic url class unit test into eight smaller (albeit ugly for now) pieces</li>
|
||||
<li>added a unit test for the new 'append random text' converter</li>
|
||||
<li>cleaned up some misc URL Class code</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import base64
|
|||
import datetime
|
||||
import hashlib
|
||||
import html
|
||||
import random
|
||||
import re
|
||||
import typing
|
||||
import urllib.parse
|
||||
|
|
@ -31,6 +32,7 @@ STRING_CONVERSION_INTEGER_ADDITION = 11
|
|||
STRING_CONVERSION_DATE_ENCODE = 12
|
||||
STRING_CONVERSION_HASH_FUNCTION = 13
|
||||
STRING_CONVERSION_DATEPARSER_DECODE = 14
|
||||
STRING_CONVERSION_APPEND_RANDOM = 15
|
||||
|
||||
conversion_type_str_lookup = {}
|
||||
|
||||
|
|
@ -38,6 +40,7 @@ conversion_type_str_lookup[ STRING_CONVERSION_REMOVE_TEXT_FROM_BEGINNING ] = 're
|
|||
conversion_type_str_lookup[ STRING_CONVERSION_REMOVE_TEXT_FROM_END ] = 'remove text from end of string'
|
||||
conversion_type_str_lookup[ STRING_CONVERSION_PREPEND_TEXT ] = 'prepend text'
|
||||
conversion_type_str_lookup[ STRING_CONVERSION_APPEND_TEXT ] = 'append text'
|
||||
conversion_type_str_lookup[ STRING_CONVERSION_APPEND_RANDOM ] = 'append random text'
|
||||
conversion_type_str_lookup[ STRING_CONVERSION_ENCODE ] = 'encode'
|
||||
conversion_type_str_lookup[ STRING_CONVERSION_DECODE ] = 'decode'
|
||||
conversion_type_str_lookup[ STRING_CONVERSION_CLIP_TEXT_FROM_BEGINNING ] = 'take the start of the string'
|
||||
|
|
@ -176,6 +179,12 @@ class StringConverter( StringProcessingStep ):
|
|||
|
||||
s = s + text
|
||||
|
||||
elif conversion_type == STRING_CONVERSION_APPEND_RANDOM:
|
||||
|
||||
( population_text, num_chars ) = data
|
||||
|
||||
s = s + ''.join( random.choices( population_text, k = num_chars ) )
|
||||
|
||||
elif conversion_type == STRING_CONVERSION_ENCODE:
|
||||
|
||||
encode_type = data
|
||||
|
|
@ -418,6 +427,12 @@ class StringConverter( StringProcessingStep ):
|
|||
|
||||
return 'append with "' + data + '"'
|
||||
|
||||
elif conversion_type == STRING_CONVERSION_APPEND_RANDOM:
|
||||
|
||||
( population_text, num_chars ) = data
|
||||
|
||||
return f'append with {HydrusData.ToHumanInt( num_chars )} random characters, from "{population_text}"'
|
||||
|
||||
elif conversion_type == STRING_CONVERSION_ENCODE:
|
||||
|
||||
return 'encode to ' + data
|
||||
|
|
|
|||
|
|
@ -10437,6 +10437,35 @@ class DB( HydrusDB.HydrusDB ):
|
|||
self.pub_initial_message( message )
|
||||
|
||||
|
||||
try:
|
||||
|
||||
domain_manager = self.modules_serialisable.GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER )
|
||||
|
||||
domain_manager.Initialise()
|
||||
|
||||
#
|
||||
|
||||
domain_manager.OverwriteDefaultParsers( [
|
||||
'gelbooru 0.2.0 file page parser'
|
||||
] )
|
||||
|
||||
#
|
||||
|
||||
domain_manager.TryToLinkURLClassesAndParsers()
|
||||
|
||||
#
|
||||
|
||||
self.modules_serialisable.SetJSONDump( domain_manager )
|
||||
|
||||
except Exception as e:
|
||||
|
||||
HydrusData.PrintException( e )
|
||||
|
||||
message = 'Trying to update some downloaders failed! Please let hydrus dev know!'
|
||||
|
||||
self.pub_initial_message( message )
|
||||
|
||||
|
||||
|
||||
self._controller.frame_splash_status.SetTitleText( 'updated db to v{}'.format( HydrusData.ToHumanInt( version + 1 ) ) )
|
||||
|
||||
|
|
|
|||
|
|
@ -2213,7 +2213,7 @@ class FrameGUI( CAC.ApplicationCommandProcessorMixin, ClientGUITopLevelWindows.M
|
|||
additional_service_keys_to_tags = ClientTags.ServiceKeysToTags()
|
||||
|
||||
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, ephemeral_ok = True )
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, for_server = True )
|
||||
|
||||
( url_type, match_name, can_parse, cannot_parse_reason ) = self._controller.network_engine.domain_manager.GetURLParseCapability( url )
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import os
|
||||
import typing
|
||||
import urllib.parse
|
||||
|
||||
from qtpy import QtWidgets as QW
|
||||
from qtpy import QtGui as QG
|
||||
|
|
@ -13,6 +14,7 @@ from hydrus.core import HydrusSerialisable
|
|||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientDefaults
|
||||
from hydrus.client import ClientGlobals as CG
|
||||
from hydrus.client import ClientParsing
|
||||
from hydrus.client import ClientStrings
|
||||
from hydrus.client.gui import ClientGUIDialogs
|
||||
from hydrus.client.gui import ClientGUIDialogsMessage
|
||||
|
|
@ -366,7 +368,7 @@ class EditGUGPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
example_url = gug.GetExampleURL()
|
||||
|
||||
example_url = CG.client_controller.network_engine.domain_manager.NormaliseURL( example_url, ephemeral_ok = True )
|
||||
example_url = CG.client_controller.network_engine.domain_manager.NormaliseURL( example_url, for_server = True )
|
||||
|
||||
self._example_url.setText( example_url )
|
||||
|
||||
|
|
@ -707,7 +709,7 @@ class EditGUGsPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
try:
|
||||
|
||||
example_url = CG.client_controller.network_engine.domain_manager.NormaliseURL( example_url, ephemeral_ok = True )
|
||||
example_url = CG.client_controller.network_engine.domain_manager.NormaliseURL( example_url, for_server = True )
|
||||
|
||||
url_class = CG.client_controller.network_engine.domain_manager.GetURLClass( example_url )
|
||||
|
||||
|
|
@ -914,6 +916,129 @@ class EditGUGsPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
|
||||
|
||||
class EditURLClassComponentPanel( ClientGUIScrolledPanels.EditPanel ):
|
||||
|
||||
def __init__( self, parent: QW.QWidget, string_match: ClientStrings.StringMatch, default_value: typing.Optional[ str ] ):
|
||||
|
||||
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
|
||||
|
||||
from hydrus.client.gui import ClientGUIStringPanels
|
||||
|
||||
string_match_panel = ClientGUICommon.StaticBox( self, 'value test' )
|
||||
|
||||
self._string_match = ClientGUIStringPanels.EditStringMatchPanel( string_match_panel, string_match )
|
||||
self._string_match.setToolTip( 'If the encoded value of the component matches this, the URL Class matches!' )
|
||||
|
||||
self._pretty_default_value = ClientGUICommon.NoneableTextCtrl( self )
|
||||
self._pretty_default_value.setToolTip( 'If the URL is missing this component, you can add it here, and the URL Class will still match and will normalise by adding this default value. This can be useful if you need to add a /art or similar to a URL that ends with either /username or /username/art--sometimes it is better to make that stuff explicit in all cases.' )
|
||||
|
||||
self._default_value = ClientGUICommon.NoneableTextCtrl( self )
|
||||
self._default_value.setToolTip( 'What actual value will be embedded into the URL sent to the server.' )
|
||||
|
||||
#
|
||||
|
||||
self.SetValue( string_match, default_value )
|
||||
|
||||
#
|
||||
|
||||
st = ClientGUICommon.BetterStaticText( string_match_panel, label = 'The String Match here will test against the value in the normalised, _%-encoded_ URL. If you have "post%20images", test for that, not "post images".' )
|
||||
st.setWordWrap( True )
|
||||
|
||||
string_match_panel.Add( st, CC.FLAGS_EXPAND_PERPENDICULAR )
|
||||
string_match_panel.Add( self._string_match, CC.FLAGS_EXPAND_BOTH_WAYS )
|
||||
|
||||
rows = []
|
||||
|
||||
rows.append( string_match_panel )
|
||||
rows.append( ( 'default value: ', self._pretty_default_value ) )
|
||||
rows.append( ( 'default value, %-encoded: ', self._default_value ) )
|
||||
|
||||
gridbox = ClientGUICommon.WrapInGrid( self, rows, add_stretch_at_end = False, expand_single_widgets = True )
|
||||
|
||||
vbox = QP.VBoxLayout()
|
||||
|
||||
QP.AddToLayout( vbox, gridbox, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
|
||||
|
||||
self.widget().setLayout( vbox )
|
||||
|
||||
self._pretty_default_value.valueChanged.connect( self._PrettyDefaultValueChanged )
|
||||
self._default_value.valueChanged.connect( self._DefaultValueChanged )
|
||||
|
||||
|
||||
def _DefaultValueChanged( self ):
|
||||
|
||||
default_value = self._default_value.GetValue()
|
||||
|
||||
pretty_default_value = default_value if default_value is None else urllib.parse.unquote( default_value )
|
||||
|
||||
self._pretty_default_value.blockSignals( True )
|
||||
|
||||
self._pretty_default_value.SetValue( pretty_default_value )
|
||||
|
||||
self._pretty_default_value.blockSignals( False )
|
||||
|
||||
|
||||
def _GetValue( self ):
|
||||
|
||||
string_match = self._string_match.GetValue()
|
||||
default_value = self._default_value.GetValue()
|
||||
|
||||
return ( string_match, default_value )
|
||||
|
||||
|
||||
def _PrettyDefaultValueChanged( self ):
|
||||
|
||||
pretty_default_value = self._pretty_default_value.GetValue()
|
||||
|
||||
default_value = pretty_default_value if pretty_default_value is None else urllib.parse.quote( pretty_default_value )
|
||||
|
||||
self._default_value.blockSignals( True )
|
||||
|
||||
self._default_value.SetValue( default_value )
|
||||
|
||||
self._default_value.blockSignals( False )
|
||||
|
||||
|
||||
def GetValue( self ):
|
||||
|
||||
( string_match, default_value ) = self._GetValue()
|
||||
|
||||
if default_value is not None and not string_match.Matches( default_value ):
|
||||
|
||||
raise HydrusExceptions.VetoException( 'That default value does not match the rule!' )
|
||||
|
||||
|
||||
return ( string_match, default_value )
|
||||
|
||||
|
||||
def SetValue( self, string_match: ClientStrings.StringMatch, default_value: typing.Optional[ str ] ):
|
||||
|
||||
self._default_value.blockSignals( True )
|
||||
|
||||
if default_value is None:
|
||||
|
||||
self._default_value.SetValue( default_value )
|
||||
|
||||
else:
|
||||
|
||||
try:
|
||||
|
||||
self._default_value.SetValue( default_value )
|
||||
|
||||
except:
|
||||
|
||||
self._default_value.SetValue( default_value )
|
||||
|
||||
|
||||
|
||||
self._default_value.blockSignals( False )
|
||||
|
||||
self._DefaultValueChanged()
|
||||
|
||||
self._string_match.SetValue( string_match )
|
||||
|
||||
|
||||
|
||||
class EditURLClassParameterFixedNamePanel( ClientGUIScrolledPanels.EditPanel ):
|
||||
|
||||
def __init__( self, parent: QW.QWidget, parameter: ClientNetworkingURLClass.URLClassParameterFixedName, dupe_names ):
|
||||
|
|
@ -924,18 +1049,36 @@ class EditURLClassParameterFixedNamePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._dupe_names = dupe_names
|
||||
|
||||
self._fixed_name = QW.QLineEdit( self )
|
||||
self._fixed_name.setToolTip( 'The "key" of the key=value pair.' )
|
||||
self._pretty_name = QW.QLineEdit( self )
|
||||
self._pretty_name.setToolTip( 'The "key" of the key=value pair.' )
|
||||
|
||||
value_string_match_panel = ClientGUICommon.StaticBox( self, 'value' )
|
||||
self._name = QW.QLineEdit( self )
|
||||
self._name.setToolTip( 'The "key" of the key=value pair. This encoded form is what is actually sent to the server!' )
|
||||
|
||||
value_string_match_panel = ClientGUICommon.StaticBox( self, 'value test' )
|
||||
|
||||
from hydrus.client.gui import ClientGUIStringPanels
|
||||
|
||||
self._value_string_match = ClientGUIStringPanels.EditStringMatchPanel( value_string_match_panel, parameter.GetValueStringMatch() )
|
||||
self._value_string_match.setToolTip( 'If the value of the key=value pair matches this, the URL Class matches!' )
|
||||
self._value_string_match.setToolTip( 'If the encoded value of the key=value pair matches this, the URL Class matches!' )
|
||||
|
||||
self._is_ephemeral = QW.QCheckBox( self )
|
||||
tt = 'THIS IS ADVANCED, DO NOT SET IF YOU ARE UNSURE! If this parameter is a one-time token or similar needed for the server request but not something you want to keep or use to compare, you can define it here.'
|
||||
tt += '\n' * 2
|
||||
tt += 'These tokens are also allowed _en masse_ in the main URL Class by setting "allow extra parameters for server", BUT if you need a whitelist, you will want to define them here. Also, if you need to pass this token on to an API/redirect converter, you have to define it here!'
|
||||
self._is_ephemeral.setToolTip( tt )
|
||||
|
||||
self._pretty_default_value = ClientGUICommon.NoneableTextCtrl( self )
|
||||
self._pretty_default_value.setToolTip( 'If the URL is missing this key=value pair, you can add it here, and the URL Class will still match and will normalise with this default value. This can be useful for gallery URLs that have an implicit page=1 or index=0 for their first result--sometimes it is better to make that stuff explicit in all cases.' )
|
||||
|
||||
self._default_value = ClientGUICommon.NoneableTextCtrl( self )
|
||||
self._default_value.setToolTip( 'If the URL is missing this key=value pair, you can add it here, and the URL Class will still match and will normalise with this default value. This can be useful for gallery URLs that have an implicit page=1 or index=0 for their first result--sometimes it is better to make that stuff explicit.' )
|
||||
self._default_value.setToolTip( 'What actual value will be embedded into the URL sent to the server.' )
|
||||
|
||||
self._default_value_string_processor = ClientGUIStringControls.StringProcessorButton( self, parameter.GetDefaultValueStringProcessor(), self._GetTestData )
|
||||
tt = 'WARNING WARNING: Extremely Big Brain'
|
||||
tt += '/n' * 2
|
||||
tt += 'You can apply the parsing system\'s normal String Processor steps to your fixed default value here. For instance, you could append/replace the default value with random hex or today\'s date. This is obviously super advanced, so be careful.'
|
||||
self._default_value_string_processor.setToolTip( tt )
|
||||
|
||||
#
|
||||
|
||||
|
|
@ -943,13 +1086,21 @@ class EditURLClassParameterFixedNamePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
#
|
||||
|
||||
st = ClientGUICommon.BetterStaticText( value_string_match_panel, label = 'The String Match here will test against the value in the normalised, _%-encoded_ URL. If you have "type=%E3%83%9D%E3%82%B9%E3%83%88", test for that, not "ポスト".' )
|
||||
st.setWordWrap( True )
|
||||
|
||||
value_string_match_panel.Add( st, CC.FLAGS_EXPAND_PERPENDICULAR )
|
||||
value_string_match_panel.Add( self._value_string_match, CC.FLAGS_EXPAND_BOTH_WAYS )
|
||||
|
||||
rows = []
|
||||
|
||||
rows.append( ( 'name: ', self._fixed_name ) )
|
||||
rows.append( ( 'name: ', self._pretty_name ) )
|
||||
rows.append( ( 'name, %-encoded: ', self._name ) )
|
||||
rows.append( value_string_match_panel )
|
||||
rows.append( ( 'default value: ', self._default_value ) )
|
||||
rows.append( ( 'is ephemeral token?: ', self._is_ephemeral ) )
|
||||
rows.append( ( 'default value: ', self._pretty_default_value ) )
|
||||
rows.append( ( 'default value, %-encoded: ', self._default_value ) )
|
||||
rows.append( ( 'default value string processor: ', self._default_value_string_processor ) )
|
||||
|
||||
gridbox = ClientGUICommon.WrapInGrid( self, rows, add_stretch_at_end = False, expand_single_widgets = True )
|
||||
|
||||
|
|
@ -959,23 +1110,113 @@ class EditURLClassParameterFixedNamePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self.widget().setLayout( vbox )
|
||||
|
||||
self._pretty_name.textChanged.connect( self._PrettyNameChanged )
|
||||
self._name.textChanged.connect( self._NameChanged )
|
||||
self._pretty_default_value.valueChanged.connect( self._PrettyDefaultValueChanged )
|
||||
self._default_value.valueChanged.connect( self._DefaultValueChanged )
|
||||
|
||||
self._is_ephemeral.clicked.connect( self._UpdateProcessorEnabled )
|
||||
self._pretty_default_value.valueChanged.connect( self._UpdateProcessorEnabled )
|
||||
self._default_value.valueChanged.connect( self._UpdateProcessorEnabled )
|
||||
|
||||
|
||||
def _UpdateProcessorEnabled( self ):
|
||||
|
||||
we_out_here = self._is_ephemeral.isChecked() and self._default_value.GetValue() is not None
|
||||
|
||||
self._default_value_string_processor.setEnabled( we_out_here )
|
||||
|
||||
|
||||
def _DefaultValueChanged( self ):
|
||||
|
||||
default_value = self._default_value.GetValue()
|
||||
|
||||
pretty_default_value = default_value if default_value is None else urllib.parse.unquote( default_value )
|
||||
|
||||
self._pretty_default_value.blockSignals( True )
|
||||
|
||||
self._pretty_default_value.SetValue( pretty_default_value )
|
||||
|
||||
self._pretty_default_value.blockSignals( False )
|
||||
|
||||
|
||||
def _GetTestData( self ) -> ClientParsing.ParsingTestData:
|
||||
|
||||
default_value = self._default_value.GetValue()
|
||||
|
||||
if default_value is None:
|
||||
|
||||
default_value = 'test'
|
||||
|
||||
|
||||
return ClientParsing.ParsingTestData( {}, texts = [ default_value ] )
|
||||
|
||||
|
||||
def _GetValue( self ):
|
||||
|
||||
name = self._fixed_name.text()
|
||||
name = self._name.text()
|
||||
|
||||
value_string_match = self._value_string_match.GetValue()
|
||||
default_value = self._default_value.GetValue()
|
||||
|
||||
parameter = ClientNetworkingURLClass.URLClassParameterFixedName(
|
||||
name = name,
|
||||
value_string_match = value_string_match,
|
||||
default_value = default_value
|
||||
value_string_match = value_string_match
|
||||
)
|
||||
|
||||
is_ephemeral = self._is_ephemeral.isChecked()
|
||||
parameter.SetIsEphemeral( is_ephemeral )
|
||||
|
||||
default_value = self._default_value.GetValue()
|
||||
parameter.SetDefaultValue( default_value )
|
||||
|
||||
if is_ephemeral and default_value is not None:
|
||||
|
||||
default_value_string_processor = self._default_value_string_processor.GetValue()
|
||||
parameter.SetDefaultValueStringProcessor( default_value_string_processor )
|
||||
|
||||
|
||||
return parameter
|
||||
|
||||
|
||||
def _NameChanged( self ):
|
||||
|
||||
name = self._name.text()
|
||||
|
||||
pretty_name = name if name is None else urllib.parse.unquote( name )
|
||||
|
||||
self._pretty_name.blockSignals( True )
|
||||
|
||||
self._pretty_name.setText( pretty_name )
|
||||
|
||||
self._pretty_name.blockSignals( False )
|
||||
|
||||
|
||||
def _PrettyDefaultValueChanged( self ):
|
||||
|
||||
pretty_default_value = self._pretty_default_value.GetValue()
|
||||
|
||||
default_value = pretty_default_value if pretty_default_value is None else urllib.parse.quote( pretty_default_value )
|
||||
|
||||
self._default_value.blockSignals( True )
|
||||
|
||||
self._default_value.SetValue( default_value )
|
||||
|
||||
self._default_value.blockSignals( False )
|
||||
|
||||
|
||||
def _PrettyNameChanged( self ):
|
||||
|
||||
pretty_name = self._pretty_name.text()
|
||||
|
||||
name = pretty_name if pretty_name is None else urllib.parse.quote( pretty_name )
|
||||
|
||||
self._name.blockSignals( True )
|
||||
|
||||
self._name.setText( name )
|
||||
|
||||
self._name.blockSignals( False )
|
||||
|
||||
|
||||
def GetValue( self ):
|
||||
|
||||
parameter = self._GetValue()
|
||||
|
|
@ -997,9 +1238,52 @@ class EditURLClassParameterFixedNamePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
def SetValue( self, parameter: ClientNetworkingURLClass.URLClassParameterFixedName ):
|
||||
|
||||
self._fixed_name.setText( parameter.GetName() )
|
||||
self._name.blockSignals( True )
|
||||
|
||||
try:
|
||||
|
||||
self._name.setText( parameter.GetName() )
|
||||
|
||||
except:
|
||||
|
||||
self._name.setText( parameter.GetName() )
|
||||
|
||||
|
||||
self._name.blockSignals( False )
|
||||
|
||||
self._NameChanged()
|
||||
|
||||
default_value = parameter.GetDefaultValue()
|
||||
|
||||
self._default_value.blockSignals( True )
|
||||
|
||||
if default_value is None:
|
||||
|
||||
self._default_value.SetValue( default_value )
|
||||
|
||||
else:
|
||||
|
||||
try:
|
||||
|
||||
self._default_value.SetValue( default_value )
|
||||
|
||||
except:
|
||||
|
||||
self._default_value.SetValue( default_value )
|
||||
|
||||
|
||||
|
||||
self._default_value.blockSignals( False )
|
||||
|
||||
self._DefaultValueChanged()
|
||||
|
||||
self._value_string_match.SetValue( parameter.GetValueStringMatch() )
|
||||
self._default_value.SetValue( parameter.GetDefaultValue() )
|
||||
|
||||
self._is_ephemeral.setChecked( parameter.IsEphemeralToken() )
|
||||
|
||||
self._default_value_string_processor.SetValue( parameter.GetDefaultValueStringProcessor() )
|
||||
|
||||
self._UpdateProcessorEnabled()
|
||||
|
||||
|
||||
|
||||
|
|
@ -1134,6 +1418,12 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._no_more_parameters_than_this.setToolTip( tt )
|
||||
|
||||
self._keep_extra_parameters_for_server = QW.QCheckBox( self._options_panel )
|
||||
|
||||
tt = 'If checked, the URL not strip out undefined parameters in the normalisation process that occurs before a URL is sent to the server. In general, you probably want to keep this on, since these extra parameters can include temporary tokens and so on. Undefined parameters are removed when URLs are compared to each other (to detect dupes) or saved to the "known urls" storage in the database.'
|
||||
|
||||
self._keep_extra_parameters_for_server.setToolTip( tt )
|
||||
|
||||
self._can_produce_multiple_files = QW.QCheckBox( self._options_panel )
|
||||
|
||||
tt = 'If checked, the client will not rely on instances of this URL class to predetermine \'already in db\' or \'previously deleted\' outcomes. This is important for post types like pixiv pages (which can ultimately be manga, and represent many pages) and tweets (which can have multiple images).'
|
||||
|
|
@ -1228,18 +1518,19 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._example_url_classes = ClientGUICommon.BetterStaticText( self )
|
||||
|
||||
self._ephemeral_normalised_url = QW.QLineEdit( self )
|
||||
self._ephemeral_normalised_url.setReadOnly( True )
|
||||
self._ephemeral_normalised_url.setToolTip( 'This is what will be sent to the server.' )
|
||||
self._for_server_normalised_url = QW.QLineEdit( self )
|
||||
self._for_server_normalised_url.setReadOnly( True )
|
||||
|
||||
self._ephemeral_normalised_url.setVisible( False )
|
||||
tt = 'This is what should actually be sent to the server. It has some elements of full normalisation, but depending on your options, there may be additional, "ephemeral" data included. If you use an API/redirect, it will be that.'
|
||||
|
||||
self._for_server_normalised_url.setToolTip( tt )
|
||||
|
||||
self._normalised_url = QW.QLineEdit( self )
|
||||
self._normalised_url.setReadOnly( True )
|
||||
|
||||
tt = 'The same url can be expressed in different ways. The parameters can be reordered, and descriptive \'sugar\' like "/123456/bodysuit-samus_aran" can be altered at a later date, say to "/123456/bodysuit-green_eyes-samus_aran". In order to collapse all the different expressions of a url down to a single comparable form, the client will \'normalise\' them based on the essential definitions in their url class. Parameters will be alphebatised and non-defined elements will be removed.'
|
||||
tt += os.linesep * 2
|
||||
tt += 'All normalisation will switch to the preferred scheme (http/https). The alphabetisation of parameters and stripping out of non-defined elements will occur for all URLs except Gallery URLs or Watchable URLs that do not use an API Lookup. (In general, you can define gallery and watchable urls a little more loosely since they generally do not need to be compared, but if you will be saving it with a file or need to perform some regex conversion into an API/Redirect URL, you\'ll want a rigorously defined url class that will normalise to something reliable and pretty.)'
|
||||
tt = 'This is the fully normalised URL, which is what is saved to the database. It is used to compare to other URLs.'
|
||||
tt += '/n' * 2
|
||||
tt += 'We want to normalise to a single reliable URL because the same URL can be expressed in different ways. The parameters can be reordered, and descriptive \'sugar\' like "/123456/bodysuit-samus_aran" can be altered at a later date, say to "/123456/bodysuit-green_eyes-samus_aran". In order to collapse all the different expressions of a url down to a single comparable form, we remove any cruft and "normalise" things. The preferred scheme (http/https) will be switched to, and, typically, parameters will be alphabetised and non-defined elements will be removed.'
|
||||
|
||||
self._normalised_url.setToolTip( tt )
|
||||
|
||||
|
|
@ -1267,6 +1558,8 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
self._no_more_path_components_than_this.setChecked( url_class.NoMorePathComponentsThanThis() )
|
||||
self._no_more_parameters_than_this.setChecked( url_class.NoMoreParametersThanThis() )
|
||||
|
||||
self._keep_extra_parameters_for_server.setChecked( url_class.KeepExtraParametersForServer() )
|
||||
|
||||
self._path_components.AddDatas( path_components )
|
||||
|
||||
self._parameters.AddDatas( parameters )
|
||||
|
|
@ -1366,6 +1659,7 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
rows.append( ( 'alphabetise GET parameters when normalising?: ', self._alphabetise_get_parameters ) )
|
||||
rows.append( ( 'do not match on any extra path components?: ', self._no_more_path_components_than_this ) )
|
||||
rows.append( ( 'do not match on any extra parameters?: ', self._no_more_parameters_than_this ) )
|
||||
rows.append( ( 'keep extra parameters for server?: ', self._keep_extra_parameters_for_server ) )
|
||||
rows.append( ( 'keep fragment when normalising?: ', self._keep_fragment ) )
|
||||
rows.append( ( 'post page can produce multiple files?: ', self._can_produce_multiple_files ) )
|
||||
rows.append( ( 'associate a \'known url\' with resulting files?: ', self._should_be_associated_with_files ) )
|
||||
|
|
@ -1390,7 +1684,7 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
rows = []
|
||||
|
||||
rows.append( ( 'example url: ', self._example_url ) )
|
||||
#rows.append( ( 'url sent to the server: ', self._ephemeral_normalised_url ) )
|
||||
rows.append( ( 'request url: ', self._for_server_normalised_url ) )
|
||||
rows.append( ( 'normalised url: ', self._normalised_url ) )
|
||||
|
||||
gridbox_2 = ClientGUICommon.WrapInGrid( self, rows )
|
||||
|
|
@ -1411,6 +1705,7 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
self._alphabetise_get_parameters.clicked.connect( self._UpdateControls )
|
||||
self._no_more_path_components_than_this.clicked.connect( self._UpdateControls )
|
||||
self._no_more_parameters_than_this.clicked.connect( self._UpdateControls )
|
||||
self._keep_extra_parameters_for_server.clicked.connect( self._UpdateControls )
|
||||
self._match_subdomains.clicked.connect( self._UpdateControls )
|
||||
self._keep_matched_subdomains.clicked.connect( self._UpdateControls )
|
||||
self._keep_fragment.clicked.connect( self._UpdateControls )
|
||||
|
|
@ -1471,19 +1766,17 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
name = parameter.GetName()
|
||||
value_string_match = parameter.GetValueStringMatch()
|
||||
|
||||
pretty_name = name
|
||||
pretty_name = urllib.parse.unquote( name )
|
||||
pretty_value_string_match = value_string_match.ToString()
|
||||
|
||||
default_value = parameter.GetDefaultValue()
|
||||
|
||||
if default_value is not None:
|
||||
if parameter.HasDefaultValue():
|
||||
|
||||
pretty_value_string_match += f' (default "{default_value}")'
|
||||
pretty_value_string_match += f' (default "{urllib.parse.unquote(parameter.GetDefaultValue( with_processing = True ))}")'
|
||||
|
||||
|
||||
if parameter.IsEphemeralToken():
|
||||
|
||||
pretty_value_string_match += ' (is ephemeral)'
|
||||
pretty_value_string_match += ' (is ephemeral)'
|
||||
|
||||
|
||||
sort_name = pretty_name
|
||||
|
|
@ -1556,49 +1849,23 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
def _EditPathComponent( self, row ):
|
||||
|
||||
( string_match, default ) = row
|
||||
( string_match, default_value ) = row
|
||||
|
||||
with ClientGUITopLevelWindowsPanels.DialogEdit( self, 'edit path component' ) as dlg:
|
||||
|
||||
from hydrus.client.gui import ClientGUIStringPanels
|
||||
|
||||
panel = ClientGUIStringPanels.EditStringMatchPanel( dlg, string_match )
|
||||
panel = EditURLClassComponentPanel( dlg, string_match, default_value )
|
||||
|
||||
dlg.SetPanel( panel )
|
||||
|
||||
if dlg.exec() == QW.QDialog.Accepted:
|
||||
|
||||
new_string_match = panel.GetValue()
|
||||
( new_string_match, new_default_value ) = panel.GetValue()
|
||||
|
||||
if default is None:
|
||||
|
||||
default = ''
|
||||
|
||||
QP.CallAfter( self._UpdateControls ) # seems sometimes this doesn't kick in naturally
|
||||
|
||||
with ClientGUIDialogs.DialogTextEntry( self, 'Enter optional \'default\' value for this path component, which will be filled in if missing. Leave blank for none (recommended).', default = default, allow_blank = True ) as dlg_default:
|
||||
|
||||
if dlg_default.exec() == QW.QDialog.Accepted:
|
||||
|
||||
new_default = dlg_default.GetValue()
|
||||
|
||||
if new_default == '':
|
||||
|
||||
new_default = None
|
||||
|
||||
elif not string_match.Matches( new_default ):
|
||||
|
||||
ClientGUIDialogsMessage.ShowWarning( self, 'That default does not match the given rule! Clearing it to none!' )
|
||||
|
||||
new_default = None
|
||||
|
||||
|
||||
new_row = ( new_string_match, new_default )
|
||||
|
||||
QP.CallAfter( self._UpdateControls ) # seems sometimes this doesn't kick in naturally
|
||||
|
||||
return new_row
|
||||
|
||||
|
||||
new_row = ( new_string_match, new_default_value )
|
||||
|
||||
return new_row
|
||||
|
||||
|
||||
raise HydrusExceptions.VetoException()
|
||||
|
|
@ -1679,6 +1946,10 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
url_class.SetNoMoreParametersThanThis( no_more )
|
||||
|
||||
keep_extra_parameters_for_server = self._keep_extra_parameters_for_server.isChecked()
|
||||
|
||||
url_class.SetKeepExtraParametersForServer( keep_extra_parameters_for_server )
|
||||
|
||||
return url_class
|
||||
|
||||
|
||||
|
|
@ -1746,6 +2017,18 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._single_value_parameters_string_match.setEnabled( self._has_single_value_parameters.isChecked() )
|
||||
|
||||
nuke_keep_extra_params = self._no_more_parameters_than_this.isChecked() or self._api_lookup_converter.GetValue().MakesChanges()
|
||||
|
||||
if nuke_keep_extra_params:
|
||||
|
||||
self._keep_extra_parameters_for_server.setChecked( False )
|
||||
self._keep_extra_parameters_for_server.setEnabled( False )
|
||||
|
||||
else:
|
||||
|
||||
self._keep_extra_parameters_for_server.setEnabled( True )
|
||||
|
||||
|
||||
#
|
||||
|
||||
url_class = self._GetValue()
|
||||
|
|
@ -1761,22 +2044,7 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
self._can_produce_multiple_files.setEnabled( False )
|
||||
|
||||
|
||||
if url_class.ClippingIsAppropriate():
|
||||
|
||||
if self._match_subdomains.isChecked():
|
||||
|
||||
self._keep_matched_subdomains.setEnabled( True )
|
||||
|
||||
else:
|
||||
|
||||
self._keep_matched_subdomains.setChecked( False )
|
||||
self._keep_matched_subdomains.setEnabled( False )
|
||||
|
||||
|
||||
else:
|
||||
|
||||
self._keep_matched_subdomains.setEnabled( False )
|
||||
|
||||
self._keep_matched_subdomains.setEnabled( self._match_subdomains.isChecked() )
|
||||
|
||||
try:
|
||||
|
||||
|
|
@ -1791,21 +2059,6 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._normalised_url.setText( normalised )
|
||||
|
||||
ephemeral_normalised = url_class.Normalise( example_url, ephemeral_ok = True )
|
||||
|
||||
if ephemeral_normalised != normalised:
|
||||
|
||||
self._ephemeral_normalised_url.setText( ephemeral_normalised )
|
||||
|
||||
self._ephemeral_normalised_url.setEnabled( True )
|
||||
|
||||
else:
|
||||
|
||||
self._ephemeral_normalised_url.setText( '' )
|
||||
|
||||
self._ephemeral_normalised_url.setEnabled( False )
|
||||
|
||||
|
||||
self._referral_url_converter.SetExampleString( normalised )
|
||||
self._api_lookup_converter.SetExampleString( normalised )
|
||||
|
||||
|
|
@ -1853,6 +2106,10 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
|
||||
|
||||
for_server_normalised = url_class.Normalise( example_url, for_server = True )
|
||||
|
||||
self._for_server_normalised_url.setText( for_server_normalised )
|
||||
|
||||
try:
|
||||
|
||||
if url_class.UsesAPIURL():
|
||||
|
|
@ -1865,6 +2122,8 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
self._example_url_classes.setObjectName( 'HydrusInvalid' )
|
||||
|
||||
|
||||
self._for_server_normalised_url.setText( api_lookup_url )
|
||||
|
||||
else:
|
||||
|
||||
api_lookup_url = 'none set'
|
||||
|
|
@ -1909,7 +2168,7 @@ class EditURLClassPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
self._example_url_classes.setText( 'Example does not match - '+reason )
|
||||
self._example_url_classes.setObjectName( 'HydrusInvalid' )
|
||||
|
||||
self._ephemeral_normalised_url.clear()
|
||||
self._for_server_normalised_url.clear()
|
||||
self._normalised_url.clear()
|
||||
self._api_url.clear()
|
||||
|
||||
|
|
@ -2083,6 +2342,8 @@ class EditURLClassesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._UpdateURLClassCheckerText()
|
||||
|
||||
self._changes_made = False
|
||||
|
||||
|
||||
def _Add( self ):
|
||||
|
||||
|
|
@ -2113,12 +2374,22 @@ class EditURLClassesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._list_ctrl.AddDatas( ( url_class, ) )
|
||||
|
||||
self._changes_made = True
|
||||
|
||||
|
||||
def _ConvertDataToListCtrlTuples( self, url_class ):
|
||||
|
||||
name = url_class.GetName()
|
||||
url_type = url_class.GetURLType()
|
||||
example_url = url_class.Normalise( url_class.GetExampleURL() )
|
||||
|
||||
try:
|
||||
|
||||
example_url = url_class.Normalise( url_class.GetExampleURL() )
|
||||
|
||||
except:
|
||||
|
||||
example_url = 'DOES NOT MATCH OWN EXAMPLE URL!! ' + url_class.GetExampleURL()
|
||||
|
||||
|
||||
pretty_name = name
|
||||
pretty_url_type = HC.url_type_string_lookup[ url_type ]
|
||||
|
|
@ -2154,6 +2425,8 @@ class EditURLClassesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
edited_datas.append( url_class )
|
||||
|
||||
self._changes_made = True
|
||||
|
||||
else:
|
||||
|
||||
break
|
||||
|
|
@ -2227,6 +2500,24 @@ class EditURLClassesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
return url_classes
|
||||
|
||||
|
||||
def UserIsOKToCancel( self ):
|
||||
|
||||
if self._changes_made or self._list_ctrl.HasDoneDeletes():
|
||||
|
||||
message = 'You have made changes. Sure you are ok to cancel?'
|
||||
|
||||
result = ClientGUIDialogsQuick.GetYesNo( self, message )
|
||||
|
||||
if result != QW.QDialog.Accepted:
|
||||
|
||||
return False
|
||||
|
||||
|
||||
|
||||
return True
|
||||
|
||||
|
||||
|
||||
class EditURLClassLinksPanel( ClientGUIScrolledPanels.EditPanel ):
|
||||
|
||||
def __init__( self, parent: QW.QWidget, network_engine, url_classes, parsers, url_class_keys_to_parser_keys ):
|
||||
|
|
|
|||
|
|
@ -528,11 +528,16 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
if selected_file_seed.IsURLFileImport():
|
||||
|
||||
main_url = selected_file_seed.file_seed_data
|
||||
referral_url = selected_file_seed.GetReferralURL()
|
||||
primary_urls = sorted( selected_file_seed.GetPrimaryURLs() )
|
||||
source_urls = sorted( selected_file_seed.GetSourceURLs() )
|
||||
|
||||
if referral_url is None and len( primary_urls ) + len( source_urls ) == 0:
|
||||
for_server_url = CG.client_controller.network_engine.domain_manager.GetURLToFetch( main_url )
|
||||
|
||||
nothing_interesting_going_on = main_url == for_server_url and referral_url is None and len( primary_urls ) == 0 and len( source_urls ) == 0
|
||||
|
||||
if nothing_interesting_going_on:
|
||||
|
||||
ClientGUIMenus.AppendMenuLabel( menu, 'no additional urls' )
|
||||
|
||||
|
|
@ -540,21 +545,23 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
url_submenu = ClientGUIMenus.GenerateMenu( menu )
|
||||
|
||||
if main_url != for_server_url:
|
||||
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, f'request url: {for_server_url}', copy_text = for_server_url )
|
||||
|
||||
|
||||
if referral_url is not None:
|
||||
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, 'referral url:' )
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, referral_url )
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, f'referral url: {referral_url}', copy_text = referral_url )
|
||||
|
||||
|
||||
if len( primary_urls ) > 0:
|
||||
|
||||
ClientGUIMenus.AppendSeparator( url_submenu )
|
||||
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, 'primary urls:' )
|
||||
|
||||
for url in primary_urls:
|
||||
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, url )
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, f'primary url: {url}', copy_text = url )
|
||||
|
||||
|
||||
|
||||
|
|
@ -562,11 +569,9 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
ClientGUIMenus.AppendSeparator( url_submenu )
|
||||
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, 'source urls:' )
|
||||
|
||||
for url in source_urls:
|
||||
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, url )
|
||||
ClientGUIMenus.AppendMenuLabel( url_submenu, f'source url: {url}', copy_text = url )
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -2841,7 +2841,7 @@ class EditFileTimestampsPanel( CAC.ApplicationCommandProcessorMixin, ClientGUISc
|
|||
|
||||
deletee_timestamp_domains = [ domain for domain in self._original_domain_modified_domains.difference( current_domains ) ]
|
||||
|
||||
deletee_result_tuples = [ ( hashes, ClientTime.TimestampData( timestamp_type = HC.TIMESTAMP_TYPE_MODIFIED_DOMAIN, location = domain ) ) for ( domain, ( hashes, datetime_value_range, user_has_edited ) ) in self._domain_modified_list_ctrl_data_dict.items() if domain in deletee_timestamp_domains ]
|
||||
deletee_result_tuples = [ ( hashes, ClientTime.TimestampData( timestamp_type = HC.TIMESTAMP_TYPE_MODIFIED_DOMAIN, location = domain ), datetime_value_range.GetStepMS() ) for ( domain, ( hashes, datetime_value_range, user_has_edited ) ) in self._domain_modified_list_ctrl_data_dict.items() if domain in deletee_timestamp_domains ]
|
||||
|
||||
result_tuples.extend( deletee_result_tuples )
|
||||
|
||||
|
|
|
|||
|
|
@ -5205,7 +5205,7 @@ class ManageURLsPanel( CAC.ApplicationCommandProcessorMixin, ClientGUIScrolledPa
|
|||
|
||||
try:
|
||||
|
||||
normalised_url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, ephemeral_ok = True )
|
||||
normalised_url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, for_server = True )
|
||||
|
||||
normalised_urls.append( normalised_url )
|
||||
|
||||
|
|
@ -5463,6 +5463,26 @@ class ManageURLsPanel( CAC.ApplicationCommandProcessorMixin, ClientGUIScrolledPa
|
|||
return command_processed
|
||||
|
||||
|
||||
def UserIsOKToOK( self ):
|
||||
|
||||
current_text = self._url_input.text()
|
||||
|
||||
if current_text != '':
|
||||
|
||||
message = 'You have text still in the input! Sure you are ok to apply?'
|
||||
|
||||
result = ClientGUIDialogsQuick.GetYesNo( self, message )
|
||||
|
||||
if result != QW.QDialog.Accepted:
|
||||
|
||||
return False
|
||||
|
||||
|
||||
|
||||
return True
|
||||
|
||||
|
||||
|
||||
class RepairFileSystemPanel( ClientGUIScrolledPanels.ManagePanel ):
|
||||
|
||||
def __init__( self, parent, missing_subfolders: typing.Collection[ ClientFilesPhysical.FilesStorageSubfolder ] ):
|
||||
|
|
|
|||
|
|
@ -704,6 +704,7 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
ClientStrings.STRING_CONVERSION_CLIP_TEXT_FROM_END,
|
||||
ClientStrings.STRING_CONVERSION_PREPEND_TEXT,
|
||||
ClientStrings.STRING_CONVERSION_APPEND_TEXT,
|
||||
ClientStrings.STRING_CONVERSION_APPEND_RANDOM,
|
||||
ClientStrings.STRING_CONVERSION_ENCODE,
|
||||
ClientStrings.STRING_CONVERSION_DECODE,
|
||||
ClientStrings.STRING_CONVERSION_REVERSE,
|
||||
|
|
@ -826,6 +827,13 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._data_hash_function.SetValue( data )
|
||||
|
||||
elif conversion_type == ClientStrings.STRING_CONVERSION_APPEND_RANDOM:
|
||||
|
||||
( population, num_chars ) = data
|
||||
|
||||
self._data_text.setText( population )
|
||||
self._data_number.setValue( num_chars )
|
||||
|
||||
elif data is not None:
|
||||
|
||||
if isinstance( data, int ):
|
||||
|
|
@ -964,6 +972,19 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._data_dateparser_label.setVisible( True )
|
||||
|
||||
elif conversion_type == ClientStrings.STRING_CONVERSION_APPEND_RANDOM:
|
||||
|
||||
self._data_text_label.setVisible( True )
|
||||
self._data_text.setVisible( True )
|
||||
|
||||
self._data_number_label.setVisible( True )
|
||||
self._data_number.setVisible( True )
|
||||
|
||||
self._data_text_label.setText( 'population' )
|
||||
self._data_number_label.setText( 'number of characters' )
|
||||
|
||||
self._data_number.setMinimum( 1 )
|
||||
|
||||
elif conversion_type in ( ClientStrings.STRING_CONVERSION_PREPEND_TEXT, ClientStrings.STRING_CONVERSION_APPEND_TEXT, ClientStrings.STRING_CONVERSION_DATE_DECODE, ClientStrings.STRING_CONVERSION_DATE_ENCODE, ClientStrings.STRING_CONVERSION_REGEX_SUB ):
|
||||
|
||||
self._data_text_label.setVisible( True )
|
||||
|
|
@ -1093,6 +1114,13 @@ class EditStringConverterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
data = self._data_decoding.GetValue()
|
||||
|
||||
elif conversion_type == ClientStrings.STRING_CONVERSION_APPEND_RANDOM:
|
||||
|
||||
population = self._data_text.text()
|
||||
number_of_chars = self._data_number.value()
|
||||
|
||||
data = ( population, number_of_chars )
|
||||
|
||||
elif conversion_type in ( ClientStrings.STRING_CONVERSION_PREPEND_TEXT, ClientStrings.STRING_CONVERSION_APPEND_TEXT ):
|
||||
|
||||
data = self._data_text.text()
|
||||
|
|
|
|||
|
|
@ -65,6 +65,7 @@ class BetterListCtrl( QW.QTreeWidget ):
|
|||
self._data_to_tuples_func = data_to_tuples_func
|
||||
|
||||
self._use_simple_delete = use_simple_delete
|
||||
self._has_done_deletes = False
|
||||
self._can_delete_callback = can_delete_callback
|
||||
|
||||
self._rows_menu_callable = None
|
||||
|
|
@ -595,6 +596,8 @@ class BetterListCtrl( QW.QTreeWidget ):
|
|||
|
||||
self.columnListContentsChanged.emit()
|
||||
|
||||
self._has_done_deletes = True
|
||||
|
||||
|
||||
def DeleteSelected( self ):
|
||||
|
||||
|
|
@ -619,6 +622,8 @@ class BetterListCtrl( QW.QTreeWidget ):
|
|||
|
||||
self.columnListContentsChanged.emit()
|
||||
|
||||
self._has_done_deletes = True
|
||||
|
||||
|
||||
def EventColumnClick( self, col ):
|
||||
|
||||
|
|
@ -733,6 +738,11 @@ class BetterListCtrl( QW.QTreeWidget ):
|
|||
return data in self._data_to_indices
|
||||
|
||||
|
||||
def HasDoneDeletes( self ):
|
||||
|
||||
return self._has_done_deletes
|
||||
|
||||
|
||||
def HasOneSelected( self ):
|
||||
|
||||
return len( self.selectedItems() ) == 1
|
||||
|
|
|
|||
|
|
@ -1568,9 +1568,13 @@ class PanelPredicateSystemKnownURLsURLClass( PanelPredicateSystemSingle ):
|
|||
|
||||
def GetDefaultPredicate( self ):
|
||||
|
||||
from hydrus.client.networking import ClientNetworkingURLClass
|
||||
|
||||
operator = True
|
||||
rule_type = 'regex'
|
||||
rule = None
|
||||
rule_type = 'url_class'
|
||||
rule = ClientNetworkingURLClass.URLClass(
|
||||
name = 'safebooru post url'
|
||||
)
|
||||
description = ''
|
||||
|
||||
return ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_SYSTEM_KNOWN_URLS, ( operator, rule_type, rule, description ) )
|
||||
|
|
|
|||
|
|
@ -1682,6 +1682,11 @@ class NoneableTextCtrl( QW.QWidget ):
|
|||
self._text.setPlaceholderText( text )
|
||||
|
||||
|
||||
def setReadOnly( self, value: bool ):
|
||||
|
||||
self._text.setReadOnly( value )
|
||||
self._checkbox.setEnabled( not value )
|
||||
|
||||
def setToolTip( self, text ):
|
||||
|
||||
QW.QWidget.setToolTip( self, text )
|
||||
|
|
|
|||
|
|
@ -1166,7 +1166,7 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
try:
|
||||
|
||||
self.file_seed_data = CG.client_controller.network_engine.domain_manager.NormaliseURL( self.file_seed_data, ephemeral_ok = True )
|
||||
self.file_seed_data = CG.client_controller.network_engine.domain_manager.NormaliseURL( self.file_seed_data, for_server = True )
|
||||
self.file_seed_data_for_comparison = CG.client_controller.network_engine.domain_manager.NormaliseURL( self.file_seed_data )
|
||||
|
||||
except HydrusExceptions.URLClassException:
|
||||
|
|
|
|||
|
|
@ -118,7 +118,7 @@ class GallerySeed( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
try:
|
||||
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, ephemeral_ok = True )
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, for_server = True )
|
||||
|
||||
except HydrusExceptions.URLClassException:
|
||||
|
||||
|
|
|
|||
|
|
@ -243,7 +243,7 @@ class MultipleWatcherImport( HydrusSerialisable.SerialisableBase ):
|
|||
return None
|
||||
|
||||
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, ephemeral_ok = True )
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, for_server = True )
|
||||
|
||||
with self._lock:
|
||||
|
||||
|
|
@ -1762,7 +1762,7 @@ class WatcherImport( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
try:
|
||||
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, ephemeral_ok = True )
|
||||
url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, for_server = True )
|
||||
|
||||
except HydrusExceptions.URLClassException:
|
||||
|
||||
|
|
|
|||
|
|
@ -53,6 +53,7 @@ class HydrusServiceClientAPI( HydrusClientService ):
|
|||
root.putChild( b'add_files', add_files )
|
||||
|
||||
add_files.putChild( b'add_file', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAddFilesAddFile( self._service, self._client_requests_domain ) )
|
||||
add_files.putChild( b'clear_file_deletion_record', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAddFilesClearDeletedFileRecord( self._service, self._client_requests_domain ) )
|
||||
add_files.putChild( b'delete_files', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAddFilesDeleteFiles( self._service, self._client_requests_domain ) )
|
||||
add_files.putChild( b'undelete_files', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAddFilesUndeleteFiles( self._service, self._client_requests_domain ) )
|
||||
add_files.putChild( b'archive_files', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAddFilesArchiveFiles( self._service, self._client_requests_domain ) )
|
||||
|
|
|
|||
|
|
@ -1764,6 +1764,7 @@ class HydrusResourceClientAPIRestrictedAddFilesAddFile( HydrusResourceClientAPIR
|
|||
return response_context
|
||||
|
||||
|
||||
|
||||
class HydrusResourceClientAPIRestrictedAddFilesArchiveFiles( HydrusResourceClientAPIRestrictedAddFiles ):
|
||||
|
||||
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
|
||||
|
|
@ -1781,6 +1782,32 @@ class HydrusResourceClientAPIRestrictedAddFilesArchiveFiles( HydrusResourceClien
|
|||
return response_context
|
||||
|
||||
|
||||
|
||||
|
||||
class HydrusResourceClientAPIRestrictedAddFilesClearDeletedFileRecord( HydrusResourceClientAPIRestrictedAddFiles ):
|
||||
|
||||
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
|
||||
|
||||
hashes = set( ParseHashes( request ) )
|
||||
|
||||
media_results = CG.client_controller.Read( 'media_results', hashes )
|
||||
|
||||
media_results = [ media_result for media_result in media_results if CC.COMBINED_LOCAL_FILE_SERVICE_KEY in media_result.GetLocationsManager().GetDeleted() ]
|
||||
|
||||
clearee_hashes = { m.GetHash() for m in media_results }
|
||||
|
||||
content_update = ClientContentUpdates.ContentUpdate( HC.CONTENT_TYPE_FILES, HC.CONTENT_UPDATE_CLEAR_DELETE_RECORD, clearee_hashes )
|
||||
|
||||
content_update_package = ClientContentUpdates.ContentUpdatePackage.STATICCreateFromContentUpdate( CC.COMBINED_LOCAL_FILE_SERVICE_KEY, content_update )
|
||||
|
||||
CG.client_controller.Write( 'content_updates', content_update_package )
|
||||
|
||||
response_context = HydrusServerResources.ResponseContext( 200 )
|
||||
|
||||
return response_context
|
||||
|
||||
|
||||
|
||||
class HydrusResourceClientAPIRestrictedAddFilesDeleteFiles( HydrusResourceClientAPIRestrictedAddFiles ):
|
||||
|
||||
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
|
||||
|
|
@ -1848,6 +1875,7 @@ class HydrusResourceClientAPIRestrictedAddFilesUnarchiveFiles( HydrusResourceCli
|
|||
return response_context
|
||||
|
||||
|
||||
|
||||
class HydrusResourceClientAPIRestrictedAddFilesUndeleteFiles( HydrusResourceClientAPIRestrictedAddFiles ):
|
||||
|
||||
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
|
||||
|
|
@ -1858,6 +1886,13 @@ class HydrusResourceClientAPIRestrictedAddFilesUndeleteFiles( HydrusResourceClie
|
|||
|
||||
location_context.LimitToServiceTypes( CG.client_controller.services_manager.GetServiceType, ( HC.LOCAL_FILE_DOMAIN, HC.COMBINED_LOCAL_MEDIA ) )
|
||||
|
||||
media_results = CG.client_controller.Read( 'media_results', hashes )
|
||||
|
||||
# this is the only scan I have to do. all the stuff like 'can I undelete from here' and 'what does an undelete to combined local media mean' is all sorted at the db level no worries
|
||||
media_results = [ media_result for media_result in media_results if CC.COMBINED_LOCAL_FILE_SERVICE_KEY in media_result.GetLocationsManager().GetCurrent() ]
|
||||
|
||||
hashes = { media_result.GetHash() for media_result in media_results }
|
||||
|
||||
content_update = ClientContentUpdates.ContentUpdate( HC.CONTENT_TYPE_FILES, HC.CONTENT_UPDATE_UNDELETE, hashes )
|
||||
|
||||
for service_key in location_context.current_service_keys:
|
||||
|
|
@ -1872,6 +1907,7 @@ class HydrusResourceClientAPIRestrictedAddFilesUndeleteFiles( HydrusResourceClie
|
|||
return response_context
|
||||
|
||||
|
||||
|
||||
class HydrusResourceClientAPIRestrictedAddFilesGenerateHashes( HydrusResourceClientAPIRestrictedAddFiles ):
|
||||
|
||||
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
|
||||
|
|
@ -2584,6 +2620,17 @@ class HydrusResourceClientAPIRestrictedAddURLsGetURLInfo( HydrusResourceClientAP
|
|||
body_dict[ 'cannot_parse_reason' ] = cannot_parse_reason
|
||||
|
||||
|
||||
try:
|
||||
|
||||
url_to_fetch = CG.client_controller.network_engine.domain_manager.GetURLToFetch( normalised_url )
|
||||
|
||||
except Exception as e:
|
||||
|
||||
raise HydrusExceptions.BadRequestException( e )
|
||||
|
||||
|
||||
body_dict[ 'request_url' ] = url_to_fetch
|
||||
|
||||
body = Dumps( body_dict, request.preferred_mime )
|
||||
|
||||
# max age of ten minutes here
|
||||
|
|
|
|||
|
|
@ -266,7 +266,7 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
|
|||
seen_url_classes.add( api_url_class )
|
||||
|
||||
|
||||
api_url = api_url_class.Normalise( api_url, ephemeral_ok = True )
|
||||
api_url = api_url_class.Normalise( api_url, for_server = True )
|
||||
|
||||
return ( api_url_class, api_url )
|
||||
|
||||
|
|
@ -1510,7 +1510,7 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
|
|||
return True
|
||||
|
||||
|
||||
def NormaliseURL( self, url, ephemeral_ok = False ):
|
||||
def NormaliseURL( self, url, for_server = False ):
|
||||
|
||||
with self._lock:
|
||||
|
||||
|
|
@ -1539,14 +1539,14 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
else:
|
||||
|
||||
normalised_url = url_class.Normalise( url, ephemeral_ok = ephemeral_ok )
|
||||
normalised_url = url_class.Normalise( url, for_server = for_server )
|
||||
|
||||
|
||||
return normalised_url
|
||||
|
||||
|
||||
|
||||
def NormaliseURLs( self, urls: typing.Collection[ str ], ephemeral_ok = False ) -> typing.List[ str ]:
|
||||
def NormaliseURLs( self, urls: typing.Collection[ str ], for_server = False ) -> typing.List[ str ]:
|
||||
|
||||
normalised_urls = []
|
||||
|
||||
|
|
@ -1554,7 +1554,7 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
try:
|
||||
|
||||
normalised_url = self.NormaliseURL( url, ephemeral_ok = ephemeral_ok )
|
||||
normalised_url = self.NormaliseURL( url, for_server = for_server )
|
||||
|
||||
except HydrusExceptions.URLClassException:
|
||||
|
||||
|
|
|
|||
|
|
@ -160,6 +160,11 @@ def ConvertQueryTextToDict( query_text ):
|
|||
|
||||
# I no longer do this. I will encode if there is no '%' in there already, which catches cases of humans pasting/typing an URL with something human, but only if it is non-destructive
|
||||
|
||||
# Update: I still hate this a bit. I should have a parameter that says 'from human=True' and then anything we ingest should go through a normalisation( from_human = True ) wash
|
||||
# I don't like the '+' exception we have to do here, and it would be better isolated to just the initian from_human wash rather than basically every time we look at an url for normalisation
|
||||
# indeed, instead of having 'from_human' in here, I could have a 'EncodeQueryDict' that does best-attempt smart encoding from_human, once
|
||||
# this guy would then just be a glorified dict parser, great
|
||||
|
||||
param_order = []
|
||||
|
||||
query_dict = {}
|
||||
|
|
@ -184,7 +189,7 @@ def ConvertQueryTextToDict( query_text ):
|
|||
|
||||
if '%' not in value:
|
||||
|
||||
value = urllib.parse.quote( value, safe = '' )
|
||||
value = urllib.parse.quote( value, safe = '+' )
|
||||
|
||||
|
||||
single_value_parameters.append( value )
|
||||
|
|
@ -196,12 +201,12 @@ def ConvertQueryTextToDict( query_text ):
|
|||
|
||||
if '%' not in key:
|
||||
|
||||
key = urllib.parse.quote( key, safe = '' )
|
||||
key = urllib.parse.quote( key, safe = '+' )
|
||||
|
||||
|
||||
if '%' not in value:
|
||||
|
||||
value = urllib.parse.quote( value, safe = '' )
|
||||
value = urllib.parse.quote( value, safe = '+' )
|
||||
|
||||
|
||||
param_order.append( key )
|
||||
|
|
@ -291,7 +296,7 @@ def GetSearchURLs( url ):
|
|||
|
||||
try:
|
||||
|
||||
ephemeral_normalised_url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, ephemeral_ok = True )
|
||||
ephemeral_normalised_url = CG.client_controller.network_engine.domain_manager.NormaliseURL( url, for_server = True )
|
||||
|
||||
search_urls.add( ephemeral_normalised_url )
|
||||
|
||||
|
|
|
|||
|
|
@ -74,9 +74,9 @@ class URLClassParameterFixedName( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_URL_CLASS_PARAMETER_FIXED_NAME
|
||||
SERIALISABLE_NAME = 'URL Class Parameter - Fixed Name'
|
||||
SERIALISABLE_VERSION = 1
|
||||
SERIALISABLE_VERSION = 2
|
||||
|
||||
def __init__( self, name = None, value_string_match = None, default_value = None ):
|
||||
def __init__( self, name = None, value_string_match = None ):
|
||||
|
||||
if name is None:
|
||||
|
||||
|
|
@ -92,7 +92,11 @@ class URLClassParameterFixedName( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
self._name = name
|
||||
self._value_string_match = value_string_match
|
||||
self._default_value = default_value
|
||||
|
||||
self._is_ephemeral = False
|
||||
|
||||
self._default_value = None
|
||||
self._default_value_string_processor = ClientStrings.StringProcessor()
|
||||
|
||||
|
||||
def __repr__( self ):
|
||||
|
|
@ -105,20 +109,60 @@ class URLClassParameterFixedName( HydrusSerialisable.SerialisableBase ):
|
|||
def _GetSerialisableInfo( self ):
|
||||
|
||||
serialisable_value_string_match = self._value_string_match.GetSerialisableTuple()
|
||||
serialisable_default_value_string_processor = self._default_value_string_processor.GetSerialisableTuple()
|
||||
|
||||
return ( self._name, serialisable_value_string_match, self._default_value )
|
||||
return ( self._name, serialisable_value_string_match, self._is_ephemeral, self._default_value, serialisable_default_value_string_processor )
|
||||
|
||||
|
||||
def _InitialiseFromSerialisableInfo( self, serialisable_info ):
|
||||
|
||||
( self._name, serialisable_value_string_match, self._default_value ) = serialisable_info
|
||||
( self._name, serialisable_value_string_match, self._is_ephemeral, self._default_value, serialisable_default_value_string_processor ) = serialisable_info
|
||||
|
||||
self._value_string_match = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_value_string_match )
|
||||
self._default_value_string_processor = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_default_value_string_processor )
|
||||
|
||||
|
||||
def GetDefaultValue( self ):
|
||||
def _UpdateSerialisableInfo( self, version, old_serialisable_info ):
|
||||
|
||||
return self._default_value
|
||||
if version == 1:
|
||||
|
||||
( name, serialisable_value_string_match, default_value ) = old_serialisable_info
|
||||
|
||||
is_ephemeral = False
|
||||
default_value_string_processor = ClientStrings.StringConverter()
|
||||
|
||||
serialisable_default_value_string_processor = default_value_string_processor.GetSerialisableTuple()
|
||||
|
||||
new_serialisable_info = ( name, serialisable_value_string_match, is_ephemeral, default_value, serialisable_default_value_string_processor )
|
||||
|
||||
return ( 2, new_serialisable_info )
|
||||
|
||||
|
||||
|
||||
def GetDefaultValue( self, with_processing = False ) -> typing.Optional[ str ]:
|
||||
|
||||
if with_processing and self._default_value is not None:
|
||||
|
||||
try:
|
||||
|
||||
result = self._default_value_string_processor.ProcessStrings( [ self._default_value ] )
|
||||
|
||||
return result[0]
|
||||
|
||||
except:
|
||||
|
||||
return self._default_value
|
||||
|
||||
|
||||
else:
|
||||
|
||||
return self._default_value
|
||||
|
||||
|
||||
|
||||
def GetDefaultValueStringProcessor( self ) -> ClientStrings.StringProcessor:
|
||||
|
||||
return self._default_value_string_processor
|
||||
|
||||
|
||||
def GetName( self ):
|
||||
|
|
@ -131,9 +175,14 @@ class URLClassParameterFixedName( HydrusSerialisable.SerialisableBase ):
|
|||
return self._value_string_match
|
||||
|
||||
|
||||
def HasDefaultValue( self ):
|
||||
|
||||
return self._default_value is not None
|
||||
|
||||
|
||||
def IsEphemeralToken( self ):
|
||||
|
||||
return False
|
||||
return self._is_ephemeral
|
||||
|
||||
|
||||
def MustBeInOriginalURL( self ):
|
||||
|
|
@ -151,6 +200,21 @@ class URLClassParameterFixedName( HydrusSerialisable.SerialisableBase ):
|
|||
return self._value_string_match.Matches( value )
|
||||
|
||||
|
||||
def SetDefaultValue( self, default_value: typing.Optional[ str ] ):
|
||||
|
||||
self._default_value = default_value
|
||||
|
||||
|
||||
def SetDefaultValueStringProcessor( self, default_value_string_processor: ClientStrings.StringProcessor ):
|
||||
|
||||
self._default_value_string_processor = default_value_string_processor
|
||||
|
||||
|
||||
def SetIsEphemeral( self, value ):
|
||||
|
||||
self._is_ephemeral = value
|
||||
|
||||
|
||||
def TestValue( self, value ):
|
||||
|
||||
self._value_string_match.Test( value )
|
||||
|
|
@ -163,7 +227,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_URL_CLASS
|
||||
SERIALISABLE_NAME = 'URL Class'
|
||||
SERIALISABLE_VERSION = 13
|
||||
SERIALISABLE_VERSION = 14
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
|
|
@ -220,6 +284,8 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' )
|
||||
)
|
||||
|
||||
parameters.append( p )
|
||||
|
||||
|
||||
if single_value_parameters_string_match is None:
|
||||
|
||||
|
|
@ -258,6 +324,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
self._alphabetise_get_parameters = True
|
||||
self._no_more_path_components_than_this = False
|
||||
self._no_more_parameters_than_this = False
|
||||
self._keep_extra_parameters_for_server = True
|
||||
self._can_produce_multiple_files = False
|
||||
self._should_be_associated_with_files = True
|
||||
self._keep_fragment = False
|
||||
|
|
@ -278,6 +345,26 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
self._example_url = example_url
|
||||
|
||||
if self._no_more_parameters_than_this or self._api_lookup_converter.MakesChanges():
|
||||
|
||||
self._keep_extra_parameters_for_server = False
|
||||
|
||||
|
||||
|
||||
def __eq__( self, other ):
|
||||
|
||||
if isinstance( other, URLClass ):
|
||||
|
||||
return self.__hash__() == other.__hash__()
|
||||
|
||||
|
||||
return NotImplemented
|
||||
|
||||
|
||||
def __hash__( self ):
|
||||
|
||||
return ( self._name, self._url_class_key ).__hash__()
|
||||
|
||||
|
||||
def _ClipNetLoc( self, netloc ):
|
||||
|
||||
|
|
@ -300,7 +387,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return netloc
|
||||
|
||||
|
||||
def _ClipAndFleshOutPath( self, path, allow_clip = True ):
|
||||
def _ClipAndFleshOutPath( self, path: str, for_server: bool ):
|
||||
|
||||
# /post/show/1326143/akunim-anthro-armband-armwear-clothed-clothing-fem
|
||||
|
||||
|
|
@ -313,7 +400,10 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
path_components = path.split( '/' )
|
||||
|
||||
if allow_clip or len( path_components ) < len( self._path_components ):
|
||||
do_clip = self.UsesAPIURL() or not for_server
|
||||
flesh_out = len( path_components ) < len( self._path_components )
|
||||
|
||||
if do_clip or flesh_out:
|
||||
|
||||
clipped_path_components = []
|
||||
|
||||
|
|
@ -347,7 +437,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return path
|
||||
|
||||
|
||||
def _ClipAndFleshOutQuery( self, query: str, ephemeral_ok: bool, allow_clip: bool = True ):
|
||||
def _ClipAndFleshOutQuery( self, query: str, for_server: bool ):
|
||||
|
||||
( query_dict, single_value_parameters, param_order ) = ClientNetworkingFunctions.ConvertQueryTextToDict( query )
|
||||
|
||||
|
|
@ -376,16 +466,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
if not match_found:
|
||||
|
||||
default_value = parameter.GetDefaultValue()
|
||||
|
||||
if default_value is None:
|
||||
|
||||
if not parameter.IsEphemeralToken():
|
||||
|
||||
raise HydrusExceptions.URLClassException( f'Could not flesh out query--no default for {name} defined!' )
|
||||
|
||||
|
||||
else:
|
||||
if parameter.HasDefaultValue():
|
||||
|
||||
if isinstance( parameter, URLClassParameterFixedName ):
|
||||
|
||||
|
|
@ -393,7 +474,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
query_dict_keys_to_parameters[ name ] = parameter
|
||||
|
||||
query_dict[ name ] = default_value
|
||||
query_dict[ name ] = parameter.GetDefaultValue( with_processing = True )
|
||||
|
||||
param_order.append( name )
|
||||
|
||||
|
|
@ -402,7 +483,15 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
raise HydrusExceptions.URLClassException( f'Could not flesh out query--cannot figure out a fixed name for {parameter}!' )
|
||||
|
||||
|
||||
|
||||
else:
|
||||
|
||||
ok_to_be_missing = parameter.IsEphemeralToken()
|
||||
|
||||
if not ok_to_be_missing:
|
||||
|
||||
raise HydrusExceptions.URLClassException( f'Could not flesh out query--no default for {name} defined!' )
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -419,7 +508,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
if possible_parameter is None:
|
||||
|
||||
if allow_clip:
|
||||
if not ( for_server and self._keep_extra_parameters_for_server ):
|
||||
|
||||
# no matching param, discard it
|
||||
continue
|
||||
|
|
@ -427,7 +516,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
else:
|
||||
|
||||
if possible_parameter.IsEphemeralToken() and not ephemeral_ok:
|
||||
if possible_parameter.IsEphemeralToken() and not for_server:
|
||||
|
||||
continue
|
||||
|
||||
|
|
@ -445,7 +534,9 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
param_order = None
|
||||
|
||||
|
||||
if not self._has_single_value_parameters:
|
||||
we_want_single_value_params = self._has_single_value_parameters or ( for_server and self._keep_extra_parameters_for_server )
|
||||
|
||||
if not we_want_single_value_params:
|
||||
|
||||
single_value_parameters = []
|
||||
|
||||
|
|
@ -465,7 +556,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
serialisable_api_lookup_converter = self._api_lookup_converter.GetSerialisableTuple()
|
||||
serialisable_referral_url_converter = self._referral_url_converter.GetSerialisableTuple()
|
||||
|
||||
booleans = ( self._match_subdomains, self._keep_matched_subdomains, self._alphabetise_get_parameters, self._no_more_path_components_than_this, self._no_more_parameters_than_this, self._can_produce_multiple_files, self._should_be_associated_with_files, self._keep_fragment )
|
||||
booleans = ( self._match_subdomains, self._keep_matched_subdomains, self._alphabetise_get_parameters, self._no_more_path_components_than_this, self._no_more_parameters_than_this, self._keep_extra_parameters_for_server, self._can_produce_multiple_files, self._should_be_associated_with_files, self._keep_fragment )
|
||||
|
||||
return (
|
||||
serialisable_url_class_key,
|
||||
|
|
@ -510,7 +601,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
self._example_url
|
||||
) = serialisable_info
|
||||
|
||||
( self._match_subdomains, self._keep_matched_subdomains, self._alphabetise_get_parameters, self._no_more_path_components_than_this, self._no_more_parameters_than_this, self._can_produce_multiple_files, self._should_be_associated_with_files, self._keep_fragment ) = booleans
|
||||
( self._match_subdomains, self._keep_matched_subdomains, self._alphabetise_get_parameters, self._no_more_path_components_than_this, self._no_more_parameters_than_this, self._keep_extra_parameters_for_server, self._can_produce_multiple_files, self._should_be_associated_with_files, self._keep_fragment ) = booleans
|
||||
|
||||
self._url_class_key = bytes.fromhex( serialisable_url_class_key )
|
||||
self._path_components = [ ( HydrusSerialisable.CreateFromSerialisableTuple( serialisable_string_match ), default ) for ( serialisable_string_match, default ) in serialisable_path_components ]
|
||||
|
|
@ -520,6 +611,11 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
self._api_lookup_converter = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_api_lookup_converter )
|
||||
self._referral_url_converter = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_referral_url_converter )
|
||||
|
||||
if self._no_more_parameters_than_this or self._api_lookup_converter.MakesChanges():
|
||||
|
||||
self._keep_extra_parameters_for_server = False
|
||||
|
||||
|
||||
|
||||
def _UpdateSerialisableInfo( self, version, old_serialisable_info ):
|
||||
|
||||
|
|
@ -757,23 +853,73 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
example_url
|
||||
) = old_serialisable_info
|
||||
|
||||
def encode_fixed_string_match( s_m: ClientStrings.StringMatch ) -> ClientStrings.StringMatch:
|
||||
|
||||
( match_type, match_value, min_chars, max_chars, example_string ) = s_m.ToTuple()
|
||||
|
||||
if match_type == ClientStrings.STRING_MATCH_FIXED:
|
||||
|
||||
match_value = urllib.parse.quote( match_value )
|
||||
example_string = urllib.parse.quote( example_string )
|
||||
|
||||
s_m = ClientStrings.StringMatch(
|
||||
match_type = match_type,
|
||||
match_value = match_value,
|
||||
min_chars = min_chars,
|
||||
max_chars = max_chars,
|
||||
example_string = example_string
|
||||
)
|
||||
|
||||
|
||||
return s_m
|
||||
|
||||
|
||||
new_parameters = HydrusSerialisable.SerialisableList()
|
||||
|
||||
for ( name, ( serialisable_value_string_match, default_value ) ) in serialisable_parameters:
|
||||
|
||||
# we are converting from post[id] to post%5Bid%5D
|
||||
name = urllib.parse.quote( name )
|
||||
|
||||
value_string_match = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_value_string_match )
|
||||
|
||||
value_string_match = encode_fixed_string_match( value_string_match )
|
||||
|
||||
parameter = URLClassParameterFixedName(
|
||||
name = name,
|
||||
value_string_match = value_string_match,
|
||||
default_value = default_value
|
||||
value_string_match = value_string_match
|
||||
)
|
||||
|
||||
if default_value is not None:
|
||||
|
||||
default_value = urllib.parse.quote( default_value )
|
||||
|
||||
parameter.SetDefaultValue( default_value )
|
||||
|
||||
|
||||
new_parameters.append( parameter )
|
||||
|
||||
|
||||
serialisable_parameters = new_parameters.GetSerialisableTuple()
|
||||
|
||||
path_components = [ ( HydrusSerialisable.CreateFromSerialisableTuple( serialisable_string_match ), default ) for ( serialisable_string_match, default ) in serialisable_path_components ]
|
||||
|
||||
new_path_components = []
|
||||
|
||||
for ( string_match, default ) in path_components:
|
||||
|
||||
string_match = encode_fixed_string_match( string_match )
|
||||
|
||||
if default is not None:
|
||||
|
||||
default = urllib.parse.quote( default )
|
||||
|
||||
|
||||
new_path_components.append( ( string_match, default ) )
|
||||
|
||||
|
||||
serialisable_path_components = [ ( string_match.GetSerialisableTuple(), default ) for ( string_match, default ) in new_path_components ]
|
||||
|
||||
new_serialisable_info = (
|
||||
serialisable_url_class_key,
|
||||
url_type,
|
||||
|
|
@ -797,6 +943,64 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return ( 13, new_serialisable_info )
|
||||
|
||||
|
||||
if version == 13:
|
||||
|
||||
(
|
||||
serialisable_url_class_key,
|
||||
url_type,
|
||||
preferred_scheme,
|
||||
netloc,
|
||||
booleans,
|
||||
serialisable_path_components,
|
||||
serialisable_parameters,
|
||||
has_single_value_parameters,
|
||||
serialisable_single_value_parameters_match,
|
||||
serialisable_header_overrides,
|
||||
serialisable_api_lookup_converter,
|
||||
send_referral_url,
|
||||
serialisable_referrel_url_converter,
|
||||
gallery_index_type,
|
||||
gallery_index_identifier,
|
||||
gallery_index_delta,
|
||||
example_url
|
||||
) = old_serialisable_info
|
||||
|
||||
( match_subdomains, keep_matched_subdomains, alphabetise_get_parameters, no_more_path_components_than_this, no_more_parameters_than_this, can_produce_multiple_files, should_be_associated_with_files, keep_fragment ) = booleans
|
||||
|
||||
api_lookup_converter = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_api_lookup_converter )
|
||||
|
||||
keep_extra_parameters_for_server = True
|
||||
|
||||
if no_more_parameters_than_this or api_lookup_converter.MakesChanges() or url_type not in ( HC.URL_TYPE_GALLERY, HC.URL_TYPE_WATCHABLE ):
|
||||
|
||||
keep_extra_parameters_for_server = False
|
||||
|
||||
|
||||
booleans = ( match_subdomains, keep_matched_subdomains, alphabetise_get_parameters, no_more_path_components_than_this, no_more_parameters_than_this, keep_extra_parameters_for_server, can_produce_multiple_files, should_be_associated_with_files, keep_fragment )
|
||||
|
||||
new_serialisable_info = (
|
||||
serialisable_url_class_key,
|
||||
url_type,
|
||||
preferred_scheme,
|
||||
netloc,
|
||||
booleans,
|
||||
serialisable_path_components,
|
||||
serialisable_parameters,
|
||||
has_single_value_parameters,
|
||||
serialisable_single_value_parameters_match,
|
||||
serialisable_header_overrides,
|
||||
serialisable_api_lookup_converter,
|
||||
send_referral_url,
|
||||
serialisable_referrel_url_converter,
|
||||
gallery_index_type,
|
||||
gallery_index_identifier,
|
||||
gallery_index_delta,
|
||||
example_url
|
||||
)
|
||||
|
||||
return ( 14, new_serialisable_info )
|
||||
|
||||
|
||||
|
||||
def AlphabetiseGetParameters( self ):
|
||||
|
||||
|
|
@ -825,11 +1029,6 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return is_a_gallery_page or is_a_multipost_post_page
|
||||
|
||||
|
||||
def ClippingIsAppropriate( self ):
|
||||
|
||||
return self._should_be_associated_with_files or self.UsesAPIURL()
|
||||
|
||||
|
||||
def GetAPILookupConverter( self ):
|
||||
|
||||
return self._api_lookup_converter
|
||||
|
|
@ -842,7 +1041,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
url = self._example_url
|
||||
|
||||
|
||||
url = self.Normalise( url, ephemeral_ok = True )
|
||||
url = self.Normalise( url, for_server = True )
|
||||
|
||||
return self._api_lookup_converter.Convert( url )
|
||||
|
||||
|
|
@ -879,7 +1078,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
def GetNextGalleryPage( self, url ):
|
||||
|
||||
url = self.Normalise( url, ephemeral_ok = True )
|
||||
url = self.Normalise( url, for_server = True )
|
||||
|
||||
p = ClientNetworkingFunctions.ParseURL( url )
|
||||
|
||||
|
|
@ -1047,12 +1246,12 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
num_required_path_components = len( [ 1 for ( string_match, default ) in self._path_components if default is None ] )
|
||||
num_total_path_components = len( self._path_components )
|
||||
num_required_parameters = len( [ 1 for parameter in self._parameters if parameter.GetDefaultValue() is None ] )
|
||||
num_required_parameters = len( [ 1 for parameter in self._parameters if not parameter.HasDefaultValue() ] )
|
||||
num_total_parameters = len( self._parameters )
|
||||
|
||||
try:
|
||||
|
||||
len_example_url = len( self.Normalise( self._example_url, ephemeral_ok = True ) )
|
||||
len_example_url = len( self.Normalise( self._example_url, for_server = True ) )
|
||||
|
||||
except:
|
||||
|
||||
|
|
@ -1092,6 +1291,11 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return self._url_type == HC.URL_TYPE_WATCHABLE
|
||||
|
||||
|
||||
def KeepExtraParametersForServer( self ):
|
||||
|
||||
return self._keep_extra_parameters_for_server
|
||||
|
||||
|
||||
def Matches( self, url ):
|
||||
|
||||
try:
|
||||
|
|
@ -1111,7 +1315,7 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return self._match_subdomains
|
||||
|
||||
|
||||
def Normalise( self, url, ephemeral_ok = False ):
|
||||
def Normalise( self, url, for_server = False ):
|
||||
|
||||
p = ClientNetworkingFunctions.ParseURL( url )
|
||||
|
||||
|
|
@ -1127,18 +1331,9 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
fragment = ''
|
||||
|
||||
|
||||
if self.ClippingIsAppropriate():
|
||||
|
||||
netloc = self._ClipNetLoc( p.netloc )
|
||||
path = self._ClipAndFleshOutPath( p.path )
|
||||
query = self._ClipAndFleshOutQuery( p.query, ephemeral_ok )
|
||||
|
||||
else:
|
||||
|
||||
netloc = p.netloc
|
||||
path = self._ClipAndFleshOutPath( p.path, allow_clip = False )
|
||||
query = self._ClipAndFleshOutQuery( p.query, ephemeral_ok, allow_clip = False )
|
||||
|
||||
netloc = self._ClipNetLoc( p.netloc )
|
||||
path = self._ClipAndFleshOutPath( p.path, for_server )
|
||||
query = self._ClipAndFleshOutQuery( p.query, for_server )
|
||||
|
||||
r = urllib.parse.ParseResult( scheme, netloc, path, params, query, fragment )
|
||||
|
||||
|
|
@ -1184,6 +1379,11 @@ class URLClass( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
self._example_url = example_url
|
||||
|
||||
|
||||
def SetKeepExtraParametersForServer( self, value ):
|
||||
|
||||
self._keep_extra_parameters_for_server = value
|
||||
|
||||
|
||||
def SetNoMorePathComponentsThanThis( self, no_more: bool ):
|
||||
|
||||
self._no_more_path_components_than_this = no_more
|
||||
|
|
|
|||
|
|
@ -105,8 +105,8 @@ options = {}
|
|||
# Misc
|
||||
|
||||
NETWORK_VERSION = 20
|
||||
SOFTWARE_VERSION = 567
|
||||
CLIENT_API_VERSION = 62
|
||||
SOFTWARE_VERSION = 568
|
||||
CLIENT_API_VERSION = 63
|
||||
|
||||
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )
|
||||
|
||||
|
|
|
|||
|
|
@ -209,6 +209,7 @@ class TestClientAPI( unittest.TestCase ):
|
|||
|
||||
expected_result = {}
|
||||
|
||||
expected_result[ 'request_url' ] = normalised_url
|
||||
expected_result[ 'normalised_url' ] = normalised_url
|
||||
expected_result[ 'url_type' ] = HC.URL_TYPE_POST
|
||||
expected_result[ 'url_type_string' ] = 'post url'
|
||||
|
|
@ -1229,6 +1230,10 @@ class TestClientAPI( unittest.TestCase ):
|
|||
|
||||
#
|
||||
|
||||
media_result = HF.GetFakeMediaResult( hash )
|
||||
|
||||
HG.test_controller.SetRead( 'media_results', [ media_result ] )
|
||||
|
||||
HG.test_controller.ClearWrites( 'content_updates' )
|
||||
|
||||
path = '/add_files/undelete_files'
|
||||
|
|
@ -1253,6 +1258,10 @@ class TestClientAPI( unittest.TestCase ):
|
|||
|
||||
#
|
||||
|
||||
media_results = [ HF.GetFakeMediaResult( h ) for h in hashes ]
|
||||
|
||||
HG.test_controller.SetRead( 'media_results', media_results )
|
||||
|
||||
HG.test_controller.ClearWrites( 'content_updates' )
|
||||
|
||||
path = '/add_files/undelete_files'
|
||||
|
|
@ -1299,6 +1308,59 @@ class TestClientAPI( unittest.TestCase ):
|
|||
|
||||
#
|
||||
|
||||
media_result = HF.GetFakeMediaResult( hash )
|
||||
|
||||
|
||||
deleted_timestamp_ms = 5000000
|
||||
previously_imported_timestamp_ms = 2500000
|
||||
|
||||
deleted_to_timestamps_ms = { CC.COMBINED_LOCAL_FILE_SERVICE_KEY : deleted_timestamp_ms, CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY : deleted_timestamp_ms, CC.LOCAL_FILE_SERVICE_KEY : deleted_timestamp_ms }
|
||||
deleted_to_previously_imported_timestamp_ms = { CC.COMBINED_LOCAL_FILE_SERVICE_KEY : previously_imported_timestamp_ms, CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY : previously_imported_timestamp_ms, CC.LOCAL_FILE_SERVICE_KEY : previously_imported_timestamp_ms }
|
||||
|
||||
times_manager = ClientMediaManagers.TimesManager()
|
||||
|
||||
times_manager.SetDeletedTimestampsMS( deleted_to_timestamps_ms )
|
||||
times_manager.SetPreviouslyImportedTimestampsMS( deleted_to_previously_imported_timestamp_ms )
|
||||
|
||||
locations_manager = ClientMediaManagers.LocationsManager(
|
||||
set(),
|
||||
set( deleted_to_timestamps_ms.keys() ),
|
||||
set(),
|
||||
set(),
|
||||
times_manager,
|
||||
inbox = False,
|
||||
urls = set(),
|
||||
service_keys_to_filenames = {}
|
||||
)
|
||||
|
||||
media_result._locations_manager = locations_manager
|
||||
|
||||
HG.test_controller.SetRead( 'media_results', [ media_result ] )
|
||||
|
||||
HG.test_controller.ClearWrites( 'content_updates' )
|
||||
|
||||
path = '/add_files/clear_file_deletion_record'
|
||||
|
||||
body_dict = { 'hash' : hash.hex() }
|
||||
|
||||
body = json.dumps( body_dict )
|
||||
|
||||
connection.request( 'POST', path, body = body, headers = headers )
|
||||
|
||||
response = connection.getresponse()
|
||||
|
||||
data = response.read()
|
||||
|
||||
self.assertEqual( response.status, 200 )
|
||||
|
||||
[ ( ( content_update_package, ), kwargs ) ] = HG.test_controller.GetWrite( 'content_updates' )
|
||||
|
||||
expected_content_update_package = ClientContentUpdates.ContentUpdatePackage.STATICCreateFromContentUpdates( CC.COMBINED_LOCAL_FILE_SERVICE_KEY, [ ClientContentUpdates.ContentUpdate( HC.CONTENT_TYPE_FILES, HC.CONTENT_UPDATE_CLEAR_DELETE_RECORD, { hash } ) ] )
|
||||
|
||||
HF.compare_content_update_packages( self, content_update_package, expected_content_update_package )
|
||||
|
||||
#
|
||||
|
||||
HG.test_controller.ClearWrites( 'content_updates' )
|
||||
|
||||
path = '/add_files/archive_files'
|
||||
|
|
@ -2986,6 +3048,7 @@ class TestClientAPI( unittest.TestCase ):
|
|||
|
||||
expected_result = {}
|
||||
|
||||
expected_result[ 'request_url' ] = url
|
||||
expected_result[ 'normalised_url' ] = url
|
||||
expected_result[ 'url_type' ] = HC.URL_TYPE_UNKNOWN
|
||||
expected_result[ 'url_type_string' ] = 'unknown url'
|
||||
|
|
@ -3000,6 +3063,7 @@ class TestClientAPI( unittest.TestCase ):
|
|||
# known
|
||||
|
||||
url = 'http://8ch.net/tv/res/1846574.html'
|
||||
request_url = 'https://8ch.net/tv/res/1846574.json'
|
||||
normalised_url = 'https://8ch.net/tv/res/1846574.html'
|
||||
# http so we can test normalised is https
|
||||
|
||||
|
|
@ -3019,6 +3083,7 @@ class TestClientAPI( unittest.TestCase ):
|
|||
|
||||
expected_result = {}
|
||||
|
||||
expected_result[ 'request_url' ] = request_url
|
||||
expected_result[ 'normalised_url' ] = normalised_url
|
||||
expected_result[ 'url_type' ] = HC.URL_TYPE_WATCHABLE
|
||||
expected_result[ 'url_type_string' ] = 'watchable url'
|
||||
|
|
@ -3052,6 +3117,7 @@ class TestClientAPI( unittest.TestCase ):
|
|||
|
||||
expected_result = {}
|
||||
|
||||
expected_result[ 'request_url' ] = normalised_url
|
||||
expected_result[ 'normalised_url' ] = normalised_url
|
||||
expected_result[ 'url_type' ] = HC.URL_TYPE_POST
|
||||
expected_result[ 'url_type_string' ] = 'post url'
|
||||
|
|
|
|||
|
|
@ -224,11 +224,9 @@ class TestBandwidthManager( unittest.TestCase ):
|
|||
pass
|
||||
|
||||
|
||||
class TestNetworkingDomain( unittest.TestCase ):
|
||||
class TestURLClasses( unittest.TestCase ):
|
||||
|
||||
def test_url_classes( self ):
|
||||
|
||||
# TODO: Yo, these all suck and should be broken into separate spammy tests with more appropriate example urls and all that!
|
||||
def test_url_class_basics( self ):
|
||||
|
||||
name = 'test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
|
|
@ -280,6 +278,38 @@ class TestNetworkingDomain( unittest.TestCase ):
|
|||
self.assertEqual( url_class.GetReferralURL( good_url, referral_url ), referral_url )
|
||||
self.assertEqual( url_class.GetReferralURL( good_url, None ), None )
|
||||
|
||||
|
||||
def test_encoding( self ):
|
||||
|
||||
name = 'test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
preferred_scheme = 'https'
|
||||
netloc = 'testbooru.cx'
|
||||
|
||||
alphabetise_get_parameters = True
|
||||
match_subdomains = False
|
||||
keep_matched_subdomains = False
|
||||
can_produce_multiple_files = False
|
||||
should_be_associated_with_files = True
|
||||
keep_fragment = False
|
||||
|
||||
path_components = []
|
||||
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ) ) )
|
||||
|
||||
send_referral_url = ClientNetworkingURLClass.SEND_REFERRAL_URL_ONLY_IF_PROVIDED
|
||||
referral_url_converter = None
|
||||
gallery_index_type = None
|
||||
gallery_index_identifier = None
|
||||
gallery_index_delta = 1
|
||||
example_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
# encoding test
|
||||
|
||||
parameters = []
|
||||
|
|
@ -297,18 +327,54 @@ class TestNetworkingDomain( unittest.TestCase ):
|
|||
self.assertEqual( url_class.Normalise( unnormalised_human_url ), normalised_encoded_url )
|
||||
self.assertEqual( url_class.Normalise( normalised_encoded_url ), normalised_encoded_url )
|
||||
|
||||
|
||||
def test_defaults( self ):
|
||||
|
||||
name = 'test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
preferred_scheme = 'https'
|
||||
netloc = 'testbooru.cx'
|
||||
|
||||
alphabetise_get_parameters = True
|
||||
match_subdomains = False
|
||||
keep_matched_subdomains = False
|
||||
can_produce_multiple_files = False
|
||||
should_be_associated_with_files = True
|
||||
keep_fragment = False
|
||||
|
||||
path_components = []
|
||||
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ) ) )
|
||||
|
||||
send_referral_url = ClientNetworkingURLClass.SEND_REFERRAL_URL_ONLY_IF_PROVIDED
|
||||
referral_url_converter = None
|
||||
gallery_index_type = None
|
||||
gallery_index_identifier = None
|
||||
gallery_index_delta = 1
|
||||
example_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
#
|
||||
|
||||
good_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
# default test
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'pid', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '0' ), default_value = '0' ) )
|
||||
|
||||
p = ClientNetworkingURLClass.URLClassParameterFixedName( name = 'pid', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '0' ) )
|
||||
|
||||
p.SetDefaultValue( '0' )
|
||||
|
||||
parameters.append( p )
|
||||
|
||||
url_class = ClientNetworkingURLClass.URLClass( name, url_type = url_type, preferred_scheme = preferred_scheme, netloc = netloc, path_components = path_components, parameters = parameters, send_referral_url = send_referral_url, referral_url_converter = referral_url_converter, gallery_index_type = gallery_index_type, gallery_index_identifier = gallery_index_identifier, gallery_index_delta = gallery_index_delta, example_url = example_url )
|
||||
|
||||
|
|
@ -326,13 +392,194 @@ class TestNetworkingDomain( unittest.TestCase ):
|
|||
self.assertTrue( url_class.Matches( unnormalised_with_pid ) )
|
||||
self.assertTrue( url_class.Matches( good_url ) )
|
||||
|
||||
|
||||
def test_is_ephemeral( self ):
|
||||
|
||||
name = 'test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
preferred_scheme = 'https'
|
||||
netloc = 'testbooru.cx'
|
||||
|
||||
alphabetise_get_parameters = True
|
||||
match_subdomains = False
|
||||
keep_matched_subdomains = False
|
||||
can_produce_multiple_files = False
|
||||
should_be_associated_with_files = True
|
||||
keep_fragment = False
|
||||
|
||||
path_components = []
|
||||
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
|
||||
|
||||
send_referral_url = ClientNetworkingURLClass.SEND_REFERRAL_URL_ONLY_IF_PROVIDED
|
||||
referral_url_converter = None
|
||||
gallery_index_type = None
|
||||
gallery_index_identifier = None
|
||||
gallery_index_delta = 1
|
||||
example_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
#
|
||||
|
||||
# default test
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ) ) )
|
||||
|
||||
p = ClientNetworkingURLClass.URLClassParameterFixedName( name = 'token', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_ANY, example_string = 'abcd' ) )
|
||||
|
||||
p.SetDefaultValue( '0' )
|
||||
p.SetIsEphemeral( True )
|
||||
|
||||
parameters.append( p )
|
||||
|
||||
url_class = ClientNetworkingURLClass.URLClass( name, url_type = url_type, preferred_scheme = preferred_scheme, netloc = netloc, path_components = path_components, parameters = parameters, send_referral_url = send_referral_url, referral_url_converter = referral_url_converter, gallery_index_type = gallery_index_type, gallery_index_identifier = gallery_index_identifier, gallery_index_delta = gallery_index_delta, example_url = example_url )
|
||||
|
||||
url_class.SetURLBooleans( match_subdomains, keep_matched_subdomains, alphabetise_get_parameters, can_produce_multiple_files, should_be_associated_with_files, keep_fragment )
|
||||
|
||||
unnormalised = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
unnormalised_and_already_has = 'https://testbooru.cx/post/page.php?id=123456&s=view&token=hello'
|
||||
for_server_normalised = 'https://testbooru.cx/post/page.php?id=123456&s=view&token=0'
|
||||
normalised = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
self.assertEqual( url_class.Normalise( unnormalised, for_server = True ), for_server_normalised )
|
||||
self.assertEqual( url_class.Normalise( unnormalised ), normalised )
|
||||
|
||||
self.assertEqual( url_class.Normalise( unnormalised_and_already_has, for_server = True ), unnormalised_and_already_has )
|
||||
self.assertEqual( url_class.Normalise( unnormalised_and_already_has ), normalised )
|
||||
|
||||
self.assertTrue( url_class.Matches( unnormalised ) )
|
||||
self.assertTrue( url_class.Matches( unnormalised_and_already_has ) )
|
||||
self.assertTrue( url_class.Matches( for_server_normalised ) )
|
||||
self.assertTrue( url_class.Matches( normalised ) )
|
||||
|
||||
|
||||
def test_defaults_with_string_processor( self ):
|
||||
|
||||
name = 'test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
preferred_scheme = 'https'
|
||||
netloc = 'testbooru.cx'
|
||||
|
||||
alphabetise_get_parameters = True
|
||||
match_subdomains = False
|
||||
keep_matched_subdomains = False
|
||||
can_produce_multiple_files = False
|
||||
should_be_associated_with_files = True
|
||||
keep_fragment = False
|
||||
|
||||
path_components = []
|
||||
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
|
||||
|
||||
send_referral_url = ClientNetworkingURLClass.SEND_REFERRAL_URL_ONLY_IF_PROVIDED
|
||||
referral_url_converter = None
|
||||
gallery_index_type = None
|
||||
gallery_index_identifier = None
|
||||
gallery_index_delta = 1
|
||||
example_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
#
|
||||
|
||||
# default test
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ) ) )
|
||||
|
||||
p = ClientNetworkingURLClass.URLClassParameterFixedName( name = 'cache_reset', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_ANY, example_string = 'abcd' ) )
|
||||
|
||||
p.SetDefaultValue( '0' )
|
||||
p.SetIsEphemeral( True )
|
||||
|
||||
sp = ClientStrings.StringProcessor()
|
||||
sp.SetProcessingSteps(
|
||||
[
|
||||
ClientStrings.StringConverter(
|
||||
conversions = [
|
||||
( ClientStrings.STRING_CONVERSION_APPEND_RANDOM, ( 'a', 5 ) )
|
||||
],
|
||||
example_string = '0'
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
p.SetDefaultValueStringProcessor( sp )
|
||||
|
||||
parameters.append( p )
|
||||
|
||||
url_class = ClientNetworkingURLClass.URLClass( name, url_type = url_type, preferred_scheme = preferred_scheme, netloc = netloc, path_components = path_components, parameters = parameters, send_referral_url = send_referral_url, referral_url_converter = referral_url_converter, gallery_index_type = gallery_index_type, gallery_index_identifier = gallery_index_identifier, gallery_index_delta = gallery_index_delta, example_url = example_url )
|
||||
|
||||
url_class.SetURLBooleans( match_subdomains, keep_matched_subdomains, alphabetise_get_parameters, can_produce_multiple_files, should_be_associated_with_files, keep_fragment )
|
||||
|
||||
unnormalised = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
unnormalised_and_already_has = 'https://testbooru.cx/post/page.php?cache_reset=hello&id=123456&s=view'
|
||||
for_server_normalised = 'https://testbooru.cx/post/page.php?cache_reset=0aaaaa&id=123456&s=view'
|
||||
normalised = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
self.assertEqual( url_class.Normalise( unnormalised, for_server = True ), for_server_normalised )
|
||||
self.assertEqual( url_class.Normalise( unnormalised ), normalised )
|
||||
|
||||
self.assertEqual( url_class.Normalise( unnormalised_and_already_has, for_server = True ), unnormalised_and_already_has )
|
||||
self.assertEqual( url_class.Normalise( unnormalised_and_already_has ), normalised )
|
||||
|
||||
self.assertTrue( url_class.Matches( unnormalised ) )
|
||||
self.assertTrue( url_class.Matches( unnormalised_and_already_has ) )
|
||||
self.assertTrue( url_class.Matches( for_server_normalised ) )
|
||||
self.assertTrue( url_class.Matches( normalised ) )
|
||||
|
||||
|
||||
def test_alphabetise_params( self ):
|
||||
|
||||
name = 'test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
preferred_scheme = 'https'
|
||||
netloc = 'testbooru.cx'
|
||||
|
||||
alphabetise_get_parameters = True
|
||||
match_subdomains = False
|
||||
keep_matched_subdomains = False
|
||||
can_produce_multiple_files = False
|
||||
should_be_associated_with_files = True
|
||||
keep_fragment = False
|
||||
|
||||
path_components = []
|
||||
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ) ) )
|
||||
|
||||
send_referral_url = ClientNetworkingURLClass.SEND_REFERRAL_URL_ONLY_IF_PROVIDED
|
||||
referral_url_converter = None
|
||||
gallery_index_type = None
|
||||
gallery_index_identifier = None
|
||||
gallery_index_delta = 1
|
||||
example_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
#
|
||||
|
||||
referral_url = 'https://testbooru.cx/gallery/tags=samus_aran'
|
||||
good_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
unnormalised_good_url_1 = 'https://testbooru.cx/post/page.php?id=123456&s=view&additional_gumpf=stuff'
|
||||
unnormalised_good_url_2 = 'https://testbooru.cx/post/page.php?s=view&id=123456'
|
||||
bad_url = 'https://wew.lad/123456'
|
||||
|
||||
#
|
||||
|
||||
url_class = ClientNetworkingURLClass.URLClass( name, url_type = url_type, preferred_scheme = preferred_scheme, netloc = netloc, path_components = path_components, parameters = parameters, send_referral_url = send_referral_url, referral_url_converter = referral_url_converter, gallery_index_type = gallery_index_type, gallery_index_identifier = gallery_index_identifier, gallery_index_delta = gallery_index_delta, example_url = example_url )
|
||||
|
||||
url_class.SetURLBooleans( match_subdomains, keep_matched_subdomains, alphabetise_get_parameters, can_produce_multiple_files, should_be_associated_with_files, keep_fragment )
|
||||
|
||||
self.assertEqual( url_class.Normalise( unnormalised_good_url_2 ), good_url )
|
||||
|
||||
alphabetise_get_parameters = False
|
||||
|
||||
url_class = ClientNetworkingURLClass.URLClass( name, url_type = url_type, preferred_scheme = preferred_scheme, netloc = netloc, path_components = path_components, parameters = parameters, send_referral_url = send_referral_url, referral_url_converter = referral_url_converter, gallery_index_type = gallery_index_type, gallery_index_identifier = gallery_index_identifier, gallery_index_delta = gallery_index_delta, example_url = example_url )
|
||||
|
|
@ -341,7 +588,41 @@ class TestNetworkingDomain( unittest.TestCase ):
|
|||
|
||||
self.assertEqual( url_class.Normalise( unnormalised_good_url_2 ), unnormalised_good_url_2 )
|
||||
|
||||
|
||||
def test_referral( self ):
|
||||
|
||||
name = 'test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
preferred_scheme = 'https'
|
||||
netloc = 'testbooru.cx'
|
||||
|
||||
alphabetise_get_parameters = True
|
||||
match_subdomains = False
|
||||
keep_matched_subdomains = False
|
||||
can_produce_multiple_files = False
|
||||
should_be_associated_with_files = True
|
||||
keep_fragment = False
|
||||
|
||||
path_components = []
|
||||
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ) ) )
|
||||
|
||||
referral_url_converter = None
|
||||
gallery_index_type = None
|
||||
gallery_index_identifier = None
|
||||
gallery_index_delta = 1
|
||||
example_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
#
|
||||
|
||||
referral_url = 'https://testbooru.cx/gallery/tags=samus_aran'
|
||||
good_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
#
|
||||
|
||||
|
|
@ -384,7 +665,8 @@ class TestNetworkingDomain( unittest.TestCase ):
|
|||
self.assertEqual( url_class.GetReferralURL( good_url, referral_url ), converted_referral_url )
|
||||
self.assertEqual( url_class.GetReferralURL( good_url, None ), converted_referral_url )
|
||||
|
||||
# fragment test
|
||||
|
||||
def test_fragment( self ):
|
||||
|
||||
name = 'mega test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
|
|
@ -427,6 +709,69 @@ class TestNetworkingDomain( unittest.TestCase ):
|
|||
|
||||
self.assertEqual( url_class.Normalise( example_url ), example_url )
|
||||
|
||||
|
||||
def test_extra_params( self ):
|
||||
|
||||
unnormalised_with_extra = 'https://testbooru.cx/post/page.php?id=123456&s=view&from_tag=skirt'
|
||||
normalised_with_extra = 'https://testbooru.cx/post/page.php?from_tag=skirt&id=123456&s=view'
|
||||
normalised_without_extra = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
name = 'test'
|
||||
url_type = HC.URL_TYPE_POST
|
||||
preferred_scheme = 'https'
|
||||
netloc = 'testbooru.cx'
|
||||
|
||||
alphabetise_get_parameters = True
|
||||
match_subdomains = False
|
||||
keep_matched_subdomains = False
|
||||
can_produce_multiple_files = False
|
||||
should_be_associated_with_files = True
|
||||
keep_fragment = False
|
||||
|
||||
path_components = []
|
||||
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'post', example_string = 'post' ), None ) )
|
||||
path_components.append( ( ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'page.php', example_string = 'page.php' ), None ) )
|
||||
|
||||
parameters = []
|
||||
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 's', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FIXED, match_value = 'view', example_string = 'view' ) ) )
|
||||
parameters.append( ClientNetworkingURLClass.URLClassParameterFixedName( name = 'id', value_string_match = ClientStrings.StringMatch( match_type = ClientStrings.STRING_MATCH_FLEXIBLE, match_value = ClientStrings.NUMERIC, example_string = '123456' ) ) )
|
||||
|
||||
send_referral_url = ClientNetworkingURLClass.SEND_REFERRAL_URL_ONLY_IF_PROVIDED
|
||||
referral_url_converter = None
|
||||
gallery_index_type = None
|
||||
gallery_index_identifier = None
|
||||
gallery_index_delta = 1
|
||||
example_url = 'https://testbooru.cx/post/page.php?id=123456&s=view'
|
||||
|
||||
url_class = ClientNetworkingURLClass.URLClass( name, url_type = url_type, preferred_scheme = preferred_scheme, netloc = netloc, path_components = path_components, parameters = parameters, send_referral_url = send_referral_url, referral_url_converter = referral_url_converter, gallery_index_type = gallery_index_type, gallery_index_identifier = gallery_index_identifier, gallery_index_delta = gallery_index_delta, example_url = example_url )
|
||||
|
||||
url_class.SetURLBooleans( match_subdomains, keep_matched_subdomains, alphabetise_get_parameters, can_produce_multiple_files, should_be_associated_with_files, keep_fragment )
|
||||
|
||||
url_class.SetKeepExtraParametersForServer( False )
|
||||
|
||||
self.assertEqual( url_class.Normalise( unnormalised_with_extra, for_server = True ), normalised_without_extra )
|
||||
self.assertEqual( url_class.Normalise( unnormalised_with_extra ), normalised_without_extra )
|
||||
|
||||
url_class.SetKeepExtraParametersForServer( True )
|
||||
|
||||
self.assertEqual( url_class.Normalise( unnormalised_with_extra, for_server = True ), normalised_with_extra )
|
||||
self.assertEqual( url_class.Normalise( unnormalised_with_extra ), normalised_without_extra )
|
||||
|
||||
self.assertTrue( url_class.Matches( unnormalised_with_extra ) )
|
||||
self.assertTrue( url_class.Matches( normalised_without_extra ) )
|
||||
self.assertTrue( url_class.Matches( normalised_with_extra ) )
|
||||
|
||||
|
||||
def test_single_value_params( self ):
|
||||
|
||||
send_referral_url = ClientNetworkingURLClass.SEND_REFERRAL_URL_ONLY_IF_PROVIDED
|
||||
referral_url_converter = None
|
||||
gallery_index_type = None
|
||||
gallery_index_identifier = None
|
||||
gallery_index_delta = 1
|
||||
|
||||
# single-value params test
|
||||
|
||||
single_value_good_url = 'https://testbooru.cx/post/page.php?id=123456&token&s=view'
|
||||
|
|
|
|||
|
|
@ -149,6 +149,12 @@ class TestStringConverter( unittest.TestCase ):
|
|||
|
||||
#
|
||||
|
||||
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_APPEND_RANDOM, ( 'a', 5 ) ) ] )
|
||||
|
||||
self.assertEqual( string_converter.Convert( 'bbbbb' ), 'bbbbbaaaaa' )
|
||||
|
||||
#
|
||||
|
||||
string_converter = ClientStrings.StringConverter( conversions = [ ( ClientStrings.STRING_CONVERSION_ENCODE, 'url percent encoding' ) ] )
|
||||
|
||||
self.assertEqual( string_converter.Convert( '01234 56789' ), '01234%2056789' )
|
||||
|
|
|
|||
|
|
@ -632,7 +632,7 @@ class Controller( object ):
|
|||
|
||||
def ImportURLFromAPI( self, url, filterable_tags, additional_service_keys_to_tags, destination_page_name, destination_page_key, show_destination_page ):
|
||||
|
||||
normalised_url = self.network_engine.domain_manager.NormaliseURL( url, ephemeral_ok = True )
|
||||
normalised_url = self.network_engine.domain_manager.NormaliseURL( url, for_server = True )
|
||||
|
||||
human_result_text = '"{}" URL added successfully.'.format( normalised_url )
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 3.1 KiB After Width: | Height: | Size: 3.1 KiB |
Loading…
Reference in New Issue