Version 557
This commit is contained in:
Hydrus Network Developer
parent
f46a22c855
commit
61d495326f
|
|
@ -7,6 +7,42 @@ title: Changelog
|
|||
!!! note
|
||||
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
|
||||
|
||||
## [Version 557](https://github.com/hydrusnetwork/hydrus/releases/tag/v557)
|
||||
|
||||
### misc
|
||||
|
||||
* optimised large tag filter edit UI. you can now paste 5,000 items into an empty tag filter blacklist in less than a second, and if you have a big tag filter, removing or adding one thing is now instant (previously, this stuff would lag 4 seconds or more, sometimes multiple minutes!!)
|
||||
* the ugoira 'num frames' counting method now discludes files ending in .js/.json, to catch future bundling of frame timings
|
||||
* the cbz scanning tech should now recognise cbzs with four or fewer pages
|
||||
* a legacy 'is this image all good?' check that happens on PIL-loading is now gone. this improves rendering for a variety of truncated files and clarifies some error messages (previously, this thing was just failing silently)
|
||||
* fixed the delete file pre-flight logic so users on the non-advanced delete dialog can now delete repository updates. previously, they saw the menu entry, but hitting it was a no-op
|
||||
|
||||
### better hash predicate parsing
|
||||
|
||||
* `system:hash` labels are a little different now. they'll say `system:hash (md5) is abcd...`, with the algorithm after the "hash". hash is omitted for sha256 (the hydrus default). this eases parsing
|
||||
* `system:similar to data` labels are a little different. they'll say 'distance' instead of 'max hamming', and the number and type of hashes they hold, and if they hold only pixel hashes, the distance is not stated
|
||||
* `system:hash` predicate parsing is now more flexible. you can put the hash type pretty much anywhere now.
|
||||
* `system:similar to` and `system:similar to data` predicate parsing is now more flexible. more combinations are allowed, and you can not include distance and it'll be fine
|
||||
* these three hash predicates now copy to clipboard with all their hashes explicitly enumerated, making strings that are fully parsable! this is a big step forward in a completely sealed import-export predicate parsing loop; now I have the tech set up to export a different phrase to clipboard than what you see in the label, I just need the examples of where it goes wrong. if there is a system predicate that copies to clipboard in a way that won't parse back, let me know and I'll see if I can fix it.
|
||||
* added more unit tests for this parsing
|
||||
|
||||
### documentation and cleanup
|
||||
|
||||
* wrote a guide on how to install 'Git for Windows' for the 'running from source' help. although most of the settings in its marathon 12-page install wizard can be left as default, the technical questions can be intimidating, so I've written them all out for a nice simple install. also brushed up some of the surrounding help here
|
||||
* added a warning to the regular 'installing and updating' help regarding the danger of test-running extract releases before updating (you can overwrite your database by accident)
|
||||
* thanks to a user, the filetypes help document is updated with Ugoira and CBZ info
|
||||
* all the 'HydrusFiletypeHandling' files are refactored to a new 'files' module. there's a bunch of them these days!
|
||||
* the hydrus.core.images module is moved beneath this 'files' module too
|
||||
* the file log list panel right-click menu now says 'open URLs'/'open files' locations' depending on whether you are looking at a URL import log or local HDD import log
|
||||
|
||||
### client api
|
||||
|
||||
* the `file_metadata` call now returns `filetype_forced` and, if so, also `original_mime` to talk about the new forced filetype system
|
||||
* the client api help and unit tests are updated to test this is working ok
|
||||
* fixed a typo that was causing too much work in the updated file info manager call (and was often returning 'null' results for half-cached `file_metadata` requests with `only_return_basic_information=true`)
|
||||
* thanks to a user, the `/add_urls/get_url_info` Client API call now has a cache timeout of ten minutes, and the `/add_urls/get_url_files` call now has a timeout of 30 seconds if all the files are 'already in db'. this should automatically reduce some overhead for several programs that talk to the Client API a lot about URLs
|
||||
* the client api version is now 58
|
||||
|
||||
## [Version 556](https://github.com/hydrusnetwork/hydrus/releases/tag/v556)
|
||||
|
||||
### misc
|
||||
|
|
@ -419,30 +455,3 @@ title: Changelog
|
|||
* I made a new application command to hold the file filter. I just pre-populate the UI with a dropdown with commond choices for now, but in future it could hold a customisable file filter, once, ha ha, I have some UI to actually edit one!
|
||||
* cleaned up various shortcut code
|
||||
* misc linting cleanup
|
||||
|
||||
## [Version 547](https://github.com/hydrusnetwork/hydrus/releases/tag/v547)
|
||||
|
||||
### mpv crash fixes
|
||||
|
||||
* tl;dr: mpv less crashy now
|
||||
* if mpv fails to load a file but not in an outright 'error' manner (this appears to mean a file using a rare format that a submodule of mpv can't handle), the client now recognises this has happened, either right after the first load, or, if the error takes longer to occur, a subsequent status interrogation, and makes several new steps to restore program stability: disconnecting the mpv window from all commands, freezing the scanbar, loading the default hydrus.png as emergency backstop, and making a popup to let the user know what just happened. previously, Qt would get rapidly unhappy as it asked things to draw on screen over the null-state player, particularly if you show/hid the scanbar several times, and it would, if not removed promptly from screen, typically lead to a program crash
|
||||
* furthermore, the scanbar now never interrogates the mpv window during its paint event. a mysterious interaction of C++ level objects during error state was causing the underlying instability here, and now I cannot reproduce this even if I try
|
||||
* I also hardened the mpv window's 'no-media' state. now, rather than showing 'nothing' when media is unloaded, each mpv player now actually idles on a black png lol
|
||||
* this tech will kick in for more extreme file failures, too, which have a different handler but seem to give the same detectable dump-out state
|
||||
* fixed a silent-but-for-debug-mode error while destroying damaged mpv windows right when the program is terminating
|
||||
|
||||
### misc
|
||||
|
||||
* thanks to a user, we now have import support for 'djvu' files. basically an open source PDF style format
|
||||
* fixed pasting an image into 'system:similar files', which I missed updating in last week's code cleanup!
|
||||
* a light but spammy legacy job that refreshed every search page's empty autocomplete every five minutes (to get updated system predicates/numbers) now only occurs to autocompletes on the current page. relatedly, when you switch to a search page you haven't looked at in five minutes, it triggers the same update immediately. this should save a tiny bit of idle CPU time and, more importantly, clear out the background job queue on larger-session clients
|
||||
* I _think_ I fixed some instances of the media viewer notes window initialising with a gigantic width on some OSes. if you often get a super wide notes window when you first open the media viewer, with it fixing itself when you cycle to a different file and back, let me know if things are any better
|
||||
* when you have a popup message that has a 'show x files' button, usually from a subscription, that routine now excludes files that have been deleted since the button was created. it updates its existing file count on a click, also, to how many files it actually will generate. if you click one of these buttons, delete some files, and then click it again, it should no longer produce ghost files in the new search page. I'm going to add some more tech to optionally handle the system:hash predicate in a page in similar ways, 'locking' it to the current page content and preserving file sort so it works nice with 'remove files' etc..
|
||||
* fixed a stupid typo that was swapping the 'allow non-local connections' server setting when making the interface for IPv6 hosts. there is a secondary check of all client IPs on every request, so I am confident this was not enabling non-local connections when undesired on IPv6, but it was disabling them by deploying the loopback interface when they should have been allowed! sorry for the trouble, and well done to the person who noticed this
|
||||
* while pursing an odd and rare problem where a download job can start even though it should be waiting on a login process, I cleaned some of the login code and logic, lowering the timeout for session cookie expiring from 60 to 45 minutes and smoothing out some confusing status-checking in the pre-login stage. I could never reproduce the problem, though, so if you have had this issue, please let me know more and I'll see if I can reproduce this reliably
|
||||
|
||||
### simple cleanup
|
||||
|
||||
* cleaned up some filetype parsing code that was getting a little messy, also reduced some overhead
|
||||
* unified the thumbnail/file filetype parsing a little, with better fallback states when a hydrus thumbnail happens for some reason not to be a jpeg or png
|
||||
* fixed an out of date menu reference in the 'help my media files are broke.txt' document. 'clear orphan files' is under 'file maintenance' now, not 'db maintenance'
|
||||
|
|
|
|||
|
|
@ -1563,6 +1563,7 @@ Response:
|
|||
"hash" : "4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2",
|
||||
"size" : 63405,
|
||||
"mime" : "image/jpeg",
|
||||
"filetype_forced" : false,
|
||||
"filetype_human" : "jpeg",
|
||||
"filetype_enum" : 1,
|
||||
"ext" : ".jpg",
|
||||
|
|
@ -1617,6 +1618,7 @@ Response:
|
|||
"hash" : "3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82",
|
||||
"size" : 199713,
|
||||
"mime" : "video/webm",
|
||||
"filetype_forced" : false,
|
||||
"filetype_human" : "webm",
|
||||
"filetype_enum" : 21,
|
||||
"ext" : ".webm",
|
||||
|
|
@ -1735,6 +1737,7 @@ Response:
|
|||
"hash" : "4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2",
|
||||
"size" : 63405,
|
||||
"mime" : "image/jpeg",
|
||||
"filetype_forced" : false,
|
||||
"filetype_human" : "jpeg",
|
||||
"filetype_enum" : 1,
|
||||
"ext" : ".jpg",
|
||||
|
|
@ -1750,6 +1753,7 @@ Response:
|
|||
"hash" : "3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82",
|
||||
"size" : 199713,
|
||||
"mime" : "video/webm",
|
||||
"filetype_forced" : false,
|
||||
"filetype_human" : "webm",
|
||||
"filetype_enum" : 21,
|
||||
"ext" : ".webm",
|
||||
|
|
@ -1778,6 +1782,8 @@ The `thumbnail_width` and `thumbnail_height` are a generally reliable prediction
|
|||
|
||||
If the file has a thumbnail, `blurhash` gives a base 83 encoded string of its [blurhash](https://blurha.sh/). `pixel_hash` is an SHA256 of the image's pixel data and should exactly match for pixel-identical files (it is used in the duplicate system for 'must be pixel duplicates').
|
||||
|
||||
If the file's filetype is forced by the user, `filetype_forced` becomes `true` and a second mime string, `original_mime` is added.
|
||||
|
||||
#### tags
|
||||
|
||||
The `tags` structure is similar to the [/add\_tags/add\_tags](#add_tags_add_tags) scheme, excepting that the status numbers are:
|
||||
|
|
|
|||
|
|
@ -119,7 +119,7 @@ To run the client:
|
|||
## Updating
|
||||
|
||||
!!! warning
|
||||
Hydrus is imageboard-tier software, wild and fun but unprofessional. It is written by one Anon spinning a lot of plates. Mistakes happen from time to time, usually in the update process. There are also no training wheels to stop you from accidentally overwriting your whole db if you screw around. Be careful when updating. Make backups beforehand!
|
||||
Hydrus is imageboard-tier software, wild and fun--but also unprofessional. It is written by one Anon spinning a lot of plates. Mistakes happen from time to time, usually in the update process. There are also no training wheels to stop you from accidentally overwriting your whole db if you screw around. Be careful when updating. Make backups beforehand!
|
||||
|
||||
**Hydrus does not auto-update. It will stay the same version unless you download and install a new one.**
|
||||
|
||||
|
|
@ -142,9 +142,14 @@ The update process:
|
|||
* If the client is running, close it!
|
||||
* If you maintain a backup, run it now!
|
||||
* If you use the installer, just download the new installer and run it. It should detect where the last install was and overwrite everything automatically.
|
||||
* If you extract, then just extract the new version right on top of your current install and overwrite manually.
|
||||
* If you extract, then just extract the new version right on top of your current install and overwrite manually. *It is wise to extract it straight from the archive to your install folder.*
|
||||
* Start your client or server. It may take a few minutes to update its database. I will say in the release post if it is likely to take longer.
|
||||
|
||||
??? warning "Be extremely careful making test runs of the Extract release"
|
||||
**Do not test-run the extract before copying it over your install!** Running the program anywhere will create database files in the /db/ dir, and if you then copy that once-run folder on top of your real install, you will overwrite your real database! <span class="spoiler">Of course it doesn't really matter, because you made a full backup before you started, right? :^)</span>
|
||||
|
||||
If you need to perform tests of an update, make sure you have a good backup before you start and then remember to delete any functional test extracts before extracting from the original archive once more for the actual 'install'.
|
||||
|
||||
Unless the update specifically disables or reconfigures something, all your files and tags and settings will be remembered after the update.
|
||||
|
||||
Releases typically need to update your database to their version. New releases can retroactively perform older database updates, so if the new version is v255 but your database is on v250, you generally only need to get the v255 release, and it'll do all the intervening v250->v251, v251->v252, etc... update steps in order as soon as you boot it. If you need to update from a release more than, say, ten versions older than current, see below. You might also like to skim the release posts or [changelog](changelog.md) to see what is new.
|
||||
|
|
|
|||
|
|
@ -34,6 +34,37 @@
|
|||
<div class="content">
|
||||
<h1 id="changelog"><a href="#changelog">changelog</a></h1>
|
||||
<ul>
|
||||
<li>
|
||||
<h2 id="version_557"><a href="#version_557">version 557</a></h2>
|
||||
<ul>
|
||||
<li><h3>misc</h3></li>
|
||||
<li>optimised large tag filter edit UI. you can now paste 5,000 items into an empty tag filter blacklist in less than a second, and if you have a big tag filter, removing or adding one thing is now instant (previously, this stuff would lag 4 seconds or more, sometimes multiple minutes!!)</li>
|
||||
<li>the ugoira 'num frames' counting method now discludes files ending in .js/.json, to catch future bundling of frame timings</li>
|
||||
<li>the cbz scanning tech should now recognise cbzs with four or fewer pages</li>
|
||||
<li>a legacy 'is this image all good?' check that happens on PIL-loading is now gone. this improves rendering for a variety of truncated files and clarifies some error messages (previously, this thing was just failing silently)</li>
|
||||
<li>fixed the delete file pre-flight logic so users on the non-advanced delete dialog can now delete repository updates. previously, they saw the menu entry, but hitting it was a no-op</li>
|
||||
<li><h3>better hash predicate parsing</h3></li>
|
||||
<li>`system:hash` labels are a little different now. they'll say `system:hash (md5) is abcd...`, with the algorithm after the "hash". hash is omitted for sha256 (the hydrus default). this eases parsing</li>
|
||||
<li>`system:similar to data` labels are a little different. they'll say 'distance' instead of 'max hamming', and the number and type of hashes they hold, and if they hold only pixel hashes, the distance is not stated</li>
|
||||
<li>`system:hash` predicate parsing is now more flexible. you can put the hash type pretty much anywhere now.</li>
|
||||
<li>`system:similar to` and `system:similar to data` predicate parsing is now more flexible. more combinations are allowed, and you can not include distance and it'll be fine</li>
|
||||
<li>these three hash predicates now copy to clipboard with all their hashes explicitly enumerated, making strings that are fully parsable! this is a big step forward in a completely sealed import-export predicate parsing loop; now I have the tech set up to export a different phrase to clipboard than what you see in the label, I just need the examples of where it goes wrong. if there is a system predicate that copies to clipboard in a way that won't parse back, let me know and I'll see if I can fix it.</li>
|
||||
<li>added more unit tests for this parsing</li>
|
||||
<li><h3>documentation and cleanup</h3></li>
|
||||
<li>wrote a guide on how to install 'Git for Windows' for the 'running from source' help. although most of the settings in its marathon 12-page install wizard can be left as default, the technical questions can be intimidating, so I've written them all out for a nice simple install. also brushed up some of the surrounding help here</li>
|
||||
<li>added a warning to the regular 'installing and updating' help regarding the danger of test-running extract releases before updating (you can overwrite your database by accident)</li>
|
||||
<li>thanks to a user, the filetypes help document is updated with Ugoira and CBZ info</li>
|
||||
<li>all the 'HydrusFiletypeHandling' files are refactored to a new 'files' module. there's a bunch of them these days!</li>
|
||||
<li>the hydrus.core.images module is moved beneath this 'files' module too</li>
|
||||
<li>the file log list panel right-click menu now says 'open URLs'/'open files' locations' depending on whether you are looking at a URL import log or local HDD import log</li>
|
||||
<li><h3>client api</h3></li>
|
||||
<li>the `file_metadata` call now returns `filetype_forced` and, if so, also `original_mime` to talk about the new forced filetype system</li>
|
||||
<li>the client api help and unit tests are updated to test this is working ok</li>
|
||||
<li>fixed a typo that was causing too much work in the updated file info manager call (and was often returning 'null' results for half-cached `file_metadata` requests with `only_return_basic_information=true`)</li>
|
||||
<li>thanks to a user, the `/add_urls/get_url_info` Client API call now has a cache timeout of ten minutes, and the `/add_urls/get_url_files` call now has a timeout of 30 seconds if all the files are 'already in db'. this should automatically reduce some overhead for several programs that talk to the Client API a lot about URLs</li>
|
||||
<li>the client api version is now 58</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<h2 id="version_556"><a href="#version_556">version 556</a></h2>
|
||||
<ul>
|
||||
|
|
@ -430,7 +461,7 @@
|
|||
<li>moved the new blurhash methods to a new `HydrusBlurhash` file</li>
|
||||
<li>moved various normalisation routines to a new `HydrusImageNormalisation` file</li>
|
||||
<li>moved various channel scanning and adjusting code to a new `HydrusImageColours` file</li>
|
||||
<li>moved the hydrus image files to the new 'hydrus.core.images' module</li>
|
||||
<li>moved the hydrus image files to the new 'hydrus.core.files.images' module</li>
|
||||
<li>cleaned up some image loading code</li>
|
||||
<li>deleted ancient and no-longer-used client db code regarding imageboard definitions, status texts, and more</li>
|
||||
<li>removed the ancient `OPENCV_OK` fallback code, which was only used, superfluously, in a couple of final places. OpenCV is not optional to run hydrus, server or client</li>
|
||||
|
|
|
|||
|
|
@ -25,7 +25,31 @@ There are now setup scripts that make this easy on Windows and Linux. You do not
|
|||
|
||||
=== "Windows"
|
||||
|
||||
First of all, you will need to install Python. Get 3.10 or 3.11 [here](https://www.python.org/downloads/windows/). During the install process, make sure it has something like 'Add Python to PATH' checked. This makes Python available to your Windows.
|
||||
??? info "Git for Windows"
|
||||
Git is an excellent tool for synchronising code across platforms. Instead of downloading and extracting the whole .zip every time you want to update, it allows you to just run one line and all the code updates are applied in about three seconds. You can also run special versions of the program, or test out changes I committed two minutes ago without having to wait for me to make a whole build. You don't have to, but I recommend you get it.
|
||||
|
||||
Installing it is simple, but it can be intimidating. These are a bunch of very clever tools coming over from Linux-land, and the installer has a 10+ page wizard with several technical questions. Luckily, the 'default' is broadly fine, but I'll write everything out so you can follow along. I can't promise this list will stay perfectly up to date, so let me know if there is something complex and new you don't understand. <span class="spoiler">This is also a record that I can refer to when I set up a new machine.</span>
|
||||
|
||||
- First off, get it [here](https://gitforwindows.org/). Run the installer.
|
||||
- On the first page, with checkboxes, I recommend you uncheck 'Windows Explorer Integration', with its 'Open Git xxx here' sub-checkboxes. This stuff will just be annoying for our purposes.
|
||||
- Then set your text editor. Select the one you use, and if you don't recognise anything, set 'notepad'.
|
||||
- Now we enter the meat of the wizard pages. Everything except the default console window is best left as default:
|
||||
- `Let Git decide` on using "master" as the default main branch name
|
||||
- `Git from the command line and also from 3rd-party software`
|
||||
- `Use bundled OpenSSH`
|
||||
- `Use the OpenSSL library`
|
||||
- `Checkout Windows-style, commit Unix-style line endings`
|
||||
- **(NOT DEFAULT)** `Use Windows' default console window`. Let's keep things simple, but it isn't a big deal.
|
||||
- `Fast-forward or merge`
|
||||
- `Git Credential Manager`
|
||||
- Do `Enable file system caching`/Do not `Enable symbolic links`
|
||||
- Do not enable experimental stuff
|
||||
|
||||
Git should now be installed on your system. Any new terminal window (shift+right-click on any folder and hit 'Open in terminal') now has the `git` command!
|
||||
|
||||
|
||||
First of all, you will need to install Python. Get 3.10 or 3.11 [here](https://www.python.org/downloads/windows/). During the install process, make sure it has something like 'Add Python to PATH' checked. This makes Python available everywhere in Windows.
|
||||
|
||||
|
||||
=== "Linux"
|
||||
|
||||
|
|
@ -35,9 +59,16 @@ There are now setup scripts that make this easy on Windows and Linux. You do not
|
|||
|
||||
You should already have python of about the correct version.
|
||||
|
||||
If you are already on a very new version of python, that's ok--you might need to select the 'advanced' setup later on and choose the '(t)est' options. If you are stuck on a much older version of python, try the same thing, but with the '(o)lder' options (but I can't promise it will work!).
|
||||
If you are already on a very new python, like 3.12+, that's ok--you might need to select the 'advanced' setup later on and choose the '(t)est' options. If you are stuck on a much older version of python, try the same thing, but with the '(o)lder' options (but I can't promise it will work!).
|
||||
|
||||
Then, get the hydrus source. The github repo is [https://github.com/hydrusnetwork/hydrus](https://github.com/hydrusnetwork/hydrus). If you are familiar with git, you can just clone the repo to the location you want with `git clone https://github.com/hydrusnetwork/hydrus`, but if not, then just go to the [latest release](https://github.com/hydrusnetwork/hydrus/releases/latest) and download and extract the source code .zip somewhere. Make sure the directory has write permissions (e.g. don't put it in "Program Files"). Extracting straight to a spare drive, something like "D:\Hydrus Network", is ideal.
|
||||
Then, get the hydrus source. It is best to get it with Git: make a new folder somewhere, open a terminal in it, and then enter:
|
||||
|
||||
git clone https://github.com/hydrusnetwork/hydrus
|
||||
|
||||
The whole repository will be copied to that location. If Git is not available, then just go to the [latest release](https://github.com/hydrusnetwork/hydrus/releases/latest) and download and extract the source code .zip somewhere.
|
||||
|
||||
!!! warning "Read-only install locations"
|
||||
Make sure the install directory has convenient write permissions (e.g. on Windows, don't put it in "Program Files"). Extracting straight to a spare drive, something like "D:\Hydrus Network", is ideal.
|
||||
|
||||
We will call the base extract directory, the one with 'hydrus_client.py' in it, `install_dir`.
|
||||
|
||||
|
|
@ -207,7 +238,7 @@ The first start will take a little longer (it has to compile all the code into s
|
|||
To update, you do the same thing as for the extract builds.
|
||||
|
||||
1. If you installed by extracting the source zip, then download the [latest release](https://github.com/hydrusnetwork/hydrus/releases/latest) source zip and extract it over the top of the folder you have, overwriting the existing source files.
|
||||
2. If you installed with git, then just run `git pull` as normal.
|
||||
2. If you installed with git, then just run `git pull` as normal. I have added easy 'git_pull' scripts to the install directory for your convenience (on Windows, just double-click 'git_pull.bat').
|
||||
|
||||
If you get a library version error when you try to boot, run the venv setup again. It is worth doing this anyway, every now and then, just to stay up to date.
|
||||
|
||||
|
|
@ -365,19 +396,21 @@ Almost everything you get through pip is provided as pre-compiled 'wheels' these
|
|||
- Get Visual Studio 14/whatever build tools
|
||||
- Pick a different library version
|
||||
|
||||
Option B is always the simpler. If opencv-headless as the requirements.txt specifies won't compile in Python 3.10, then try a newer version--there will probably be one of these new highly compatible wheels and it'll just work in seconds. Check my build scripts and various requirements.txts for ideas on what versions to try for your python etc...
|
||||
Option B is always simpler. If opencv-headless as the requirements.txt specifies won't compile in your python, then try a newer version--there will probably be one of these new highly compatible wheels and it'll just work in seconds. Check my build scripts and various requirements.txts for ideas on what versions to try for your python etc...
|
||||
|
||||
If you are confident you need Visual Studio tools, then prepare for headaches. Although the tools are free from Microsoft, it can be a pain to get them through the official (and often huge) downloader installer from Microsoft. Expect a 5GB+ install with an eye-watering number of checkboxes that probably needs some stackexchange searches to figure out.
|
||||
|
||||
On Windows 10, [Chocolatey](https://chocolatey.org/) has been the easy answer. Get it installed and and use this one simple line:
|
||||
On Windows 10, [Chocolatey](https://chocolatey.org/) has been the easy answer. These can be useful:
|
||||
|
||||
```
|
||||
choco install -y vcbuildtools visualstudio2017buildtools windows-sdk-10.0
|
||||
choco install -y vcredist-all
|
||||
choco install -y vcbuildtools (this is Visual Studio 2015)
|
||||
choco install -y visualstudio2017buildtools
|
||||
choco install -y visualstudio2022buildtools
|
||||
choco install -y windows-sdk-10.0
|
||||
```
|
||||
|
||||
Trust me, just do this, it will save a ton of headaches!
|
||||
|
||||
_Update:_ On Windows 11, in 2023-01, I had trouble with the above. There's a couple '11' SDKs that installed ok, but the vcbuildtools stuff had unusual errors. I hadn't done this in years, so maybe they are broken for Windows 10 too! The good news is that a basic stock Win 11 install with Python 3.10 is fine getting everything on our requirements and even making a build without any extra compiler tech.
|
||||
_Update:_ On Windows 11, I have had some trouble with the above. The VS2015 seems not to install any more. A basic stock Win 11 install with Python 3.10 or 3.11 is fine getting everything on our requirements, but freezing with PyInstaller may have trouble finding certain 'api-***.dll' files. I am now trying to figure this out with my latest dev machine as of 2024-01. If you try this, let me know what you find out!
|
||||
|
||||
### Additional Windows Info { id="additional_windows" }
|
||||
|
||||
|
|
|
|||
|
|
@ -1009,7 +1009,7 @@ class Controller( HydrusController.HydrusController ):
|
|||
|
||||
if self.new_options.GetBoolean( 'use_system_ffmpeg' ):
|
||||
|
||||
from hydrus.core import HydrusVideoHandling
|
||||
from hydrus.core.files import HydrusVideoHandling
|
||||
|
||||
if HydrusVideoHandling.FFMPEG_PATH.startswith( HC.BIN_DIR ):
|
||||
|
||||
|
|
|
|||
|
|
@ -10,9 +10,8 @@ from hydrus.core import HydrusGlobals as HG
|
|||
from hydrus.core import HydrusSerialisable
|
||||
from hydrus.core import HydrusTags
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.images import HydrusImageMetadata
|
||||
from hydrus.core.images import HydrusImageOpening
|
||||
from hydrus.core.files.images import HydrusImageMetadata
|
||||
from hydrus.core.files.images import HydrusImageOpening
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientThreading
|
||||
|
|
|
|||
|
|
@ -8,19 +8,19 @@ import typing
|
|||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPSDHandling
|
||||
from hydrus.core import HydrusLists
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusThreading
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core import HydrusVideoHandling
|
||||
from hydrus.core.images import HydrusBlurhash
|
||||
from hydrus.core.images import HydrusImageColours
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.images import HydrusImageMetadata
|
||||
from hydrus.core.images import HydrusImageOpening
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
from hydrus.core.files import HydrusPSDHandling
|
||||
from hydrus.core.files import HydrusVideoHandling
|
||||
from hydrus.core.files.images import HydrusBlurhash
|
||||
from hydrus.core.files.images import HydrusImageColours
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageMetadata
|
||||
from hydrus.core.files.images import HydrusImageOpening
|
||||
from hydrus.core.networking import HydrusNetworking
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ import cv2
|
|||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
cv_interpolation_enum_lookup = {}
|
||||
|
||||
|
|
|
|||
|
|
@ -443,7 +443,7 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
|
|||
self._dictionary[ 'integers' ][ 'max_connection_attempts_allowed' ] = 5
|
||||
self._dictionary[ 'integers' ][ 'max_request_attempts_allowed_get' ] = 5
|
||||
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
self._dictionary[ 'integers' ][ 'thumbnail_scale_type' ] = HydrusImageHandling.THUMBNAIL_SCALE_DOWN_ONLY
|
||||
|
||||
|
|
|
|||
|
|
@ -22,8 +22,8 @@ from qtpy import QtCore as QC
|
|||
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusPDFHandling
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusPDFHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from hydrus.client.gui import ClientGUIFunctions
|
||||
|
||||
|
|
|
|||
|
|
@ -7,15 +7,15 @@ import typing
|
|||
from qtpy import QtCore as QC
|
||||
from qtpy import QtGui as QG
|
||||
|
||||
from hydrus.core import HydrusAnimationHandling
|
||||
from hydrus.core import HydrusCompression
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusVideoHandling
|
||||
from hydrus.core.images import HydrusImageColours
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusAnimationHandling
|
||||
from hydrus.core.files import HydrusVideoHandling
|
||||
from hydrus.core.files.images import HydrusImageColours
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from hydrus.client import ClientFiles
|
||||
from hydrus.client import ClientImageHandling
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ from qtpy import QtGui as QG
|
|||
from qtpy import QtCore as QC
|
||||
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusSVGHandling
|
||||
from hydrus.core.files import HydrusSVGHandling
|
||||
|
||||
from hydrus.client.gui import ClientGUIFunctions
|
||||
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ from hydrus.core import HydrusGlobals as HG
|
|||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusSerialisable
|
||||
from hydrus.core import HydrusTemp
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client.gui import ClientGUIFunctions
|
||||
|
|
|
|||
|
|
@ -4,8 +4,8 @@ from hydrus.core import HydrusData
|
|||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.images import HydrusImageNormalisation
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageNormalisation
|
||||
|
||||
class GIFRenderer( object ):
|
||||
|
||||
|
|
|
|||
|
|
@ -6,13 +6,13 @@ import time
|
|||
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusThreading
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusBlurhash
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
from hydrus.core.files.images import HydrusBlurhash
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientFiles
|
||||
|
|
|
|||
|
|
@ -22,7 +22,6 @@ from hydrus.core import HydrusConstants as HC
|
|||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusEncryption
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusMemory
|
||||
from hydrus.core import HydrusPaths
|
||||
|
|
@ -32,9 +31,10 @@ from hydrus.core import HydrusTags
|
|||
from hydrus.core import HydrusTemp
|
||||
from hydrus.core import HydrusText
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core import HydrusVideoHandling
|
||||
from hydrus.core import HydrusPSDHandling
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
from hydrus.core.files import HydrusPSDHandling
|
||||
from hydrus.core.files import HydrusVideoHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.networking import HydrusNetwork
|
||||
from hydrus.core.networking import HydrusNetworking
|
||||
|
||||
|
|
|
|||
|
|
@ -295,7 +295,7 @@ def PopulateFileSeedCacheMenu( win: QW.QWidget, menu: QW.QMenu, file_seed_cache:
|
|||
|
||||
class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
|
||||
|
||||
def __init__( self, parent, controller, file_seed_cache ):
|
||||
def __init__( self, parent, controller, file_seed_cache: ClientImportFileSeeds.FileSeedCache ):
|
||||
|
||||
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
|
||||
|
||||
|
|
@ -582,7 +582,16 @@ class EditFileSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
ClientGUIMenus.AppendSeparator( menu )
|
||||
|
||||
ClientGUIMenus.AppendMenuItem( menu, 'open sources', 'Open all the selected sources in your file explorer or web browser.', self._OpenSelectedFileSeedData )
|
||||
if self._file_seed_cache.IsURLFileSeeds():
|
||||
|
||||
open_sources_text = 'open URLs'
|
||||
|
||||
else:
|
||||
|
||||
open_sources_text = 'open files\' locations'
|
||||
|
||||
|
||||
ClientGUIMenus.AppendMenuItem( menu, open_sources_text, 'Open all the selected sources in your file explorer or web browser.', self._OpenSelectedFileSeedData )
|
||||
|
||||
ClientGUIMenus.AppendSeparator( menu )
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ from hydrus.core import HydrusText
|
|||
from hydrus.client.gui import ClientGUIDialogsMessage
|
||||
from hydrus.client.gui import QtInit
|
||||
from hydrus.client.gui import QtPorting as QP
|
||||
from hydrus.core.images import HydrusImageNormalisation
|
||||
from hydrus.core.files.images import HydrusImageNormalisation
|
||||
|
||||
def ClientToScreen( win: QW.QWidget, pos: QC.QPoint ) -> QC.QPoint:
|
||||
|
||||
|
|
|
|||
|
|
@ -11,8 +11,8 @@ from hydrus.core import HydrusGlobals as HG
|
|||
from hydrus.core import HydrusLists
|
||||
from hydrus.core import HydrusThreading
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusImageMetadata
|
||||
from hydrus.core.images import HydrusImageOpening
|
||||
from hydrus.core.files.images import HydrusImageMetadata
|
||||
from hydrus.core.files.images import HydrusImageOpening
|
||||
|
||||
from hydrus.client import ClientApplicationCommand as CAC
|
||||
from hydrus.client import ClientConstants as CC
|
||||
|
|
|
|||
|
|
@ -560,11 +560,11 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._default_reason = default_reason
|
||||
|
||||
local_file_services = list( HG.client_controller.services_manager.GetServices( ( HC.LOCAL_FILE_DOMAIN, ) ) )
|
||||
local_file_service_domains = list( HG.client_controller.services_manager.GetServices( ( HC.LOCAL_FILE_DOMAIN, ) ) )
|
||||
|
||||
if suggested_file_service_key is None:
|
||||
|
||||
suggested_file_service_key = local_file_services[0].GetServiceKey()
|
||||
suggested_file_service_key = local_file_service_domains[0].GetServiceKey()
|
||||
|
||||
|
||||
self._media = self._FilterForDeleteLock( ClientMedia.FlattenMedia( media ), suggested_file_service_key )
|
||||
|
|
@ -575,7 +575,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
self._simple_description = ClientGUICommon.BetterStaticText( self, label = 'init' )
|
||||
|
||||
self._num_actionable_local_file_services = 0
|
||||
self._num_actionable_local_file_service_domains = 0
|
||||
self._permitted_action_choices = []
|
||||
self._this_dialog_includes_service_keys = False
|
||||
|
||||
|
|
@ -816,10 +816,10 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
possible_file_service_keys = []
|
||||
|
||||
local_file_services = list( HG.client_controller.services_manager.GetServices( ( HC.LOCAL_FILE_DOMAIN, ) ) )
|
||||
local_file_service_keys = { service.GetServiceKey() for service in local_file_services }
|
||||
local_file_service_domains = list( HG.client_controller.services_manager.GetServices( ( HC.LOCAL_FILE_DOMAIN, ) ) )
|
||||
local_file_service_domain_keys = { service.GetServiceKey() for service in local_file_service_domains }
|
||||
|
||||
possible_file_service_keys.extend( ( ( lfs.GetServiceKey(), lfs.GetServiceKey() ) for lfs in local_file_services ) )
|
||||
possible_file_service_keys.extend( ( ( lfs.GetServiceKey(), lfs.GetServiceKey() ) for lfs in local_file_service_domains ) )
|
||||
|
||||
possible_file_service_keys.append( ( CC.TRASH_SERVICE_KEY, CC.COMBINED_LOCAL_FILE_SERVICE_KEY ) )
|
||||
|
||||
|
|
@ -827,6 +827,11 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
possible_file_service_keys.append( ( CC.COMBINED_LOCAL_FILE_SERVICE_KEY, CC.COMBINED_LOCAL_FILE_SERVICE_KEY ) )
|
||||
|
||||
else:
|
||||
|
||||
# if not advanced, we still want regular users, in odd fixing situations, able to delete update files
|
||||
possible_file_service_keys.append( ( CC.LOCAL_UPDATE_SERVICE_KEY, CC.COMBINED_LOCAL_FILE_SERVICE_KEY ) )
|
||||
|
||||
|
||||
possible_file_service_keys.extend( ( ( rfs.GetServiceKey(), rfs.GetServiceKey() ) for rfs in HG.client_controller.services_manager.GetServices( ( HC.FILE_REPOSITORY, ) ) ) )
|
||||
|
||||
|
|
@ -842,7 +847,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
possible_file_service_keys_and_hashes = [ ( fsk, keys_to_hashes[ fsk ] ) for fsk in possible_file_service_keys if fsk in keys_to_hashes and len( keys_to_hashes[ fsk ] ) > 0 ]
|
||||
|
||||
self._num_actionable_local_file_services = len( local_file_service_keys.intersection( ( fsk[0] for ( fsk, hashes ) in possible_file_service_keys_and_hashes ) ) )
|
||||
self._num_actionable_local_file_service_domains = len( local_file_service_domain_keys.intersection( ( fsk[0] for ( fsk, hashes ) in possible_file_service_keys_and_hashes ) ) )
|
||||
|
||||
possibilities_involve_spicy_physical_delete = False
|
||||
|
||||
|
|
@ -872,7 +877,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
file_desc = '{} files'.format( HydrusData.ToHumanInt( num_to_delete ) )
|
||||
|
||||
|
||||
if self._num_actionable_local_file_services == 1:
|
||||
if self._num_actionable_local_file_service_domains == 1:
|
||||
|
||||
template = 'Send {} from {} to trash?'
|
||||
|
||||
|
|
@ -898,7 +903,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
num_local_services_done += 1
|
||||
|
||||
# this is an ugly place to put this, and the mickey-mouse append, but it works
|
||||
if self._num_actionable_local_file_services > 1 and num_local_services_done == self._num_actionable_local_file_services:
|
||||
if self._num_actionable_local_file_service_domains > 1 and num_local_services_done == self._num_actionable_local_file_service_domains:
|
||||
|
||||
self._permitted_action_choices.append( ( text, ( deletee_file_service_key, list_of_service_keys_to_content_updates, save_reason, hashes_physically_deleted, text ) ) )
|
||||
|
||||
|
|
@ -1000,7 +1005,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
unnatural_spicy_physical_delete = possibilities_involve_spicy_physical_delete and not we_are_advanced_delete_dialog
|
||||
|
||||
if self._num_actionable_local_file_services == 1 and not unnatural_spicy_physical_delete and not HC.options[ 'confirm_trash' ]:
|
||||
if self._num_actionable_local_file_service_domains == 1 and not unnatural_spicy_physical_delete and not HC.options[ 'confirm_trash' ]:
|
||||
|
||||
# this dialog will never show
|
||||
self._question_is_already_resolved = True
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ from hydrus.core import HydrusPaths
|
|||
from hydrus.core import HydrusSerialisable
|
||||
from hydrus.core import HydrusTags
|
||||
from hydrus.core import HydrusText
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from hydrus.client import ClientApplicationCommand as CAC
|
||||
from hydrus.client import ClientConstants as CC
|
||||
|
|
|
|||
|
|
@ -15,7 +15,6 @@ from hydrus.core import HydrusCompression
|
|||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusSerialisable
|
||||
|
|
@ -24,6 +23,7 @@ from hydrus.core import HydrusTagArchive
|
|||
from hydrus.core import HydrusText
|
||||
from hydrus.core import HydrusThreading
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientData
|
||||
|
|
|
|||
|
|
@ -823,57 +823,46 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
self.SetValue( tag_filter )
|
||||
|
||||
|
||||
def _AdvancedAddBlacklist( self, tag_slice ):
|
||||
|
||||
tag_slice = self._CleanTagSliceInput( tag_slice )
|
||||
|
||||
if tag_slice in self._advanced_blacklist.GetTagSlices():
|
||||
|
||||
self._advanced_blacklist.RemoveTagSlices( ( tag_slice, ) )
|
||||
|
||||
else:
|
||||
|
||||
self._advanced_whitelist.RemoveTagSlices( ( tag_slice, ) )
|
||||
|
||||
if self._CurrentlyBlocked( tag_slice ):
|
||||
|
||||
self._ShowRedundantError( HydrusTags.ConvertTagSliceToString( tag_slice ) + ' is already blocked by a broader rule!' )
|
||||
|
||||
|
||||
self._advanced_blacklist.AddTagSlices( ( tag_slice, ) )
|
||||
|
||||
|
||||
self._UpdateStatus()
|
||||
|
||||
|
||||
def _AdvancedAddBlacklistMultiple( self, tag_slices ):
|
||||
|
||||
for tag_slice in tag_slices:
|
||||
tag_slices = [ self._CleanTagSliceInput( tag_slice ) for tag_slice in tag_slices ]
|
||||
|
||||
tag_slices = HydrusData.DedupeList( tag_slices )
|
||||
|
||||
current_blacklist = set( self._advanced_blacklist.GetTagSlices() )
|
||||
|
||||
to_remove = set( tag_slices ).intersection( current_blacklist )
|
||||
|
||||
if len( to_remove ) > 0:
|
||||
|
||||
self._AdvancedAddBlacklist( tag_slice )
|
||||
self._advanced_blacklist.RemoveTagSlices( to_remove )
|
||||
|
||||
|
||||
|
||||
def _AdvancedAddWhitelist( self, tag_slice ):
|
||||
to_add = [ tag_slice for tag_slice in tag_slices if tag_slice not in to_remove ]
|
||||
|
||||
tag_slice = self._CleanTagSliceInput( tag_slice )
|
||||
|
||||
if tag_slice in self._advanced_whitelist.GetTagSlices():
|
||||
if len( to_add ) > 0:
|
||||
|
||||
self._advanced_whitelist.RemoveTagSlices( ( tag_slice, ) )
|
||||
self._advanced_whitelist.RemoveTagSlices( to_add )
|
||||
|
||||
else:
|
||||
already_blocked = [ tag_slice for tag_slice in to_add if self._CurrentlyBlocked( tag_slice ) ]
|
||||
|
||||
self._advanced_blacklist.RemoveTagSlices( ( tag_slice, ) )
|
||||
|
||||
# if it is still blocked after that, it needs whitelisting explicitly
|
||||
|
||||
if not self._CurrentlyBlocked( tag_slice ) and tag_slice not in ( '', ':' ):
|
||||
if len( already_blocked ) > 0:
|
||||
|
||||
self._ShowRedundantError( HydrusTags.ConvertTagSliceToString( tag_slice ) + ' is already permitted by a broader rule!' )
|
||||
if len( already_blocked ) == 1:
|
||||
|
||||
message = f'{HydrusTags.ConvertTagSliceToString( already_blocked[0] )} is already blocked by a broader rule!'
|
||||
|
||||
else:
|
||||
|
||||
separator = '\n' if len( already_blocked ) < 5 else ', '
|
||||
|
||||
message = 'The tags\n\n' + separator.join( [ HydrusTags.ConvertTagSliceToString( tag_slice ) for tag_slice in already_blocked ] ) + '\n\nare already blocked by a broader rule!'
|
||||
|
||||
|
||||
self._ShowRedundantError( message )
|
||||
|
||||
|
||||
self._advanced_whitelist.AddTagSlices( ( tag_slice, ) )
|
||||
self._advanced_blacklist.AddTagSlices( to_add )
|
||||
|
||||
|
||||
self._UpdateStatus()
|
||||
|
|
@ -881,11 +870,46 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
def _AdvancedAddWhitelistMultiple( self, tag_slices ):
|
||||
|
||||
for tag_slice in tag_slices:
|
||||
tag_slices = [ self._CleanTagSliceInput( tag_slice ) for tag_slice in tag_slices ]
|
||||
|
||||
current_whitelist = set( self._advanced_whitelist.GetTagSlices() )
|
||||
|
||||
to_remove = set( tag_slices ).intersection( current_whitelist )
|
||||
|
||||
if len( to_remove ) > 0:
|
||||
|
||||
self._AdvancedAddWhitelist( tag_slice )
|
||||
self._advanced_whitelist.RemoveTagSlices( to_remove )
|
||||
|
||||
|
||||
to_add = [ tag_slice for tag_slice in tag_slices if tag_slice not in to_remove ]
|
||||
|
||||
if len( to_add ) > 0:
|
||||
|
||||
self._advanced_blacklist.RemoveTagSlices( to_add )
|
||||
|
||||
already_permitted = [ tag_slice for tag_slice in to_add if tag_slice not in ( '', ':' ) and not self._CurrentlyBlocked( tag_slice ) ]
|
||||
|
||||
if len( already_permitted ) > 0:
|
||||
|
||||
if len( already_permitted ) == 1:
|
||||
|
||||
message = f'{HydrusTags.ConvertTagSliceToString( to_add[0] )} is already permitted by a broader rule!'
|
||||
|
||||
else:
|
||||
|
||||
separator = '\n' if len( already_permitted ) < 5 else ', '
|
||||
|
||||
message = 'The tags\n\n' + separator.join( [ HydrusTags.ConvertTagSliceToString( tag_slice ) for tag_slice in already_permitted ] ) + '\n\nare already permitted by a broader rule!'
|
||||
|
||||
|
||||
self._ShowRedundantError( message )
|
||||
|
||||
|
||||
self._advanced_whitelist.AddTagSlices( tag_slices )
|
||||
|
||||
|
||||
self._UpdateStatus()
|
||||
|
||||
|
||||
def _AdvancedBlacklistEverything( self ):
|
||||
|
||||
|
|
@ -1456,25 +1480,24 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
def _SimpleAddWhitelistMultiple( self, tag_slices ):
|
||||
|
||||
for tag_slice in tag_slices:
|
||||
tag_slices = set( tag_slices )
|
||||
|
||||
for simple in ( '', ':' ):
|
||||
|
||||
if tag_slice in ( '', ':' ) and tag_slice in self._simple_whitelist.GetTagSlices():
|
||||
if simple in tag_slices and simple in self._simple_whitelist.GetTagSlices():
|
||||
|
||||
self._AdvancedAddBlacklist( tag_slice )
|
||||
tag_slices.discard( simple )
|
||||
|
||||
else:
|
||||
|
||||
self._AdvancedAddWhitelist( tag_slice )
|
||||
self._AdvancedAddBlacklistMultiple( ( simple, ) )
|
||||
|
||||
|
||||
|
||||
self._AdvancedAddWhitelistMultiple( tag_slices )
|
||||
|
||||
|
||||
def _SimpleBlacklistRemoved( self, tag_slices ):
|
||||
|
||||
for tag_slice in tag_slices:
|
||||
|
||||
self._AdvancedAddBlacklist( tag_slice )
|
||||
|
||||
self._AdvancedAddBlacklistMultiple( tag_slices )
|
||||
|
||||
|
||||
def _SimpleBlacklistReset( self ):
|
||||
|
|
@ -1530,14 +1553,11 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
tag_slices.discard( simple )
|
||||
|
||||
self._AdvancedAddBlacklist( simple )
|
||||
self._AdvancedAddBlacklistMultiple( ( simple, ) )
|
||||
|
||||
|
||||
|
||||
for tag_slice in tag_slices:
|
||||
|
||||
self._AdvancedAddWhitelist( tag_slice )
|
||||
|
||||
self._AdvancedAddWhitelistMultiple( tag_slices )
|
||||
|
||||
|
||||
def _SimpleWhitelistReset( self ):
|
||||
|
|
@ -1808,7 +1828,7 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
tag_slice = self._simple_blacklist_namespace_checkboxes.GetData( index )
|
||||
|
||||
self._AdvancedAddBlacklist( tag_slice )
|
||||
self._AdvancedAddBlacklistMultiple( ( tag_slice, ) )
|
||||
|
||||
|
||||
|
||||
|
|
@ -1820,7 +1840,7 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
tag_slice = self._simple_blacklist_global_checkboxes.GetData( index )
|
||||
|
||||
self._AdvancedAddBlacklist( tag_slice )
|
||||
self._AdvancedAddBlacklistMultiple( ( tag_slice, ) )
|
||||
|
||||
|
||||
|
||||
|
|
@ -1832,12 +1852,12 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
tag_slice = self._simple_whitelist_namespace_checkboxes.GetData( index )
|
||||
|
||||
self._AdvancedAddWhitelist( tag_slice )
|
||||
self._AdvancedAddWhitelistMultiple( ( tag_slice, ) )
|
||||
|
||||
|
||||
|
||||
def EventSimpleWhitelistGlobalCheck( self, index ):
|
||||
|
||||
|
||||
index = index.row()
|
||||
|
||||
if index != -1:
|
||||
|
|
@ -1846,11 +1866,11 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
if tag_slice in ( '', ':' ) and tag_slice in self._simple_whitelist.GetTagSlices():
|
||||
|
||||
self._AdvancedAddBlacklist( tag_slice )
|
||||
self._AdvancedAddBlacklistMultiple( ( tag_slice, ) )
|
||||
|
||||
else:
|
||||
|
||||
self._AdvancedAddWhitelist( tag_slice )
|
||||
self._AdvancedAddWhitelistMultiple( ( tag_slice, ) )
|
||||
|
||||
|
||||
|
||||
|
|
@ -1859,15 +1879,8 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
tag_filter = HydrusTags.TagFilter()
|
||||
|
||||
for tag_slice in self._advanced_blacklist.GetTagSlices():
|
||||
|
||||
tag_filter.SetRule( tag_slice, HC.FILTER_BLACKLIST )
|
||||
|
||||
|
||||
for tag_slice in self._advanced_whitelist.GetTagSlices():
|
||||
|
||||
tag_filter.SetRule( tag_slice, HC.FILTER_WHITELIST )
|
||||
|
||||
tag_filter.SetRules( self._advanced_blacklist.GetTagSlices(), HC.FILTER_BLACKLIST )
|
||||
tag_filter.SetRules( self._advanced_whitelist.GetTagSlices(), HC.FILTER_WHITELIST )
|
||||
|
||||
return tag_filter
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ from hydrus.core import HydrusLists
|
|||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusTags
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from hydrus.client import ClientApplicationCommand as CAC
|
||||
from hydrus.client import ClientConstants as CC
|
||||
|
|
|
|||
|
|
@ -19,12 +19,11 @@ except:
|
|||
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPSDHandling
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from hydrus.client import ClientApplicationCommand as CAC
|
||||
from hydrus.client import ClientConstants as CC
|
||||
|
|
|
|||
|
|
@ -7,11 +7,11 @@ import typing
|
|||
from qtpy import QtCore as QC
|
||||
from qtpy import QtWidgets as QW
|
||||
|
||||
from hydrus.core import HydrusAnimationHandling
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core.files import HydrusAnimationHandling
|
||||
|
||||
from hydrus.client import ClientApplicationCommand as CAC
|
||||
from hydrus.client import ClientConstants as CC
|
||||
|
|
|
|||
|
|
@ -516,7 +516,7 @@ class ListBoxItemPredicate( ListBoxItem ):
|
|||
|
||||
else:
|
||||
|
||||
text = self._predicate.ToString( with_count = with_counts )
|
||||
text = self._predicate.ToString( with_count = with_counts, for_parsable_export = True )
|
||||
|
||||
|
||||
return text
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ from hydrus.core import HydrusExceptions
|
|||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.networking import HydrusNetwork
|
||||
|
||||
from hydrus.client import ClientApplicationCommand as CAC
|
||||
|
|
|
|||
|
|
@ -9,10 +9,10 @@ from qtpy import QtWidgets as QW
|
|||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusTemp
|
||||
from hydrus.core import HydrusText
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientParsing
|
||||
|
|
|
|||
|
|
@ -9,8 +9,8 @@ from hydrus.core import HydrusConstants as HC
|
|||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientImageHandling
|
||||
|
|
|
|||
|
|
@ -12,13 +12,13 @@ import urllib.parse
|
|||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusSerialisable
|
||||
from hydrus.core import HydrusTags
|
||||
from hydrus.core import HydrusTemp
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientData
|
||||
|
|
@ -2869,7 +2869,7 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
|
|||
return earliest_timestamp
|
||||
|
||||
|
||||
def GetExampleFileSeed( self ):
|
||||
def GetExampleURLFileSeed( self ):
|
||||
|
||||
with self._lock:
|
||||
|
||||
|
|
@ -3096,6 +3096,18 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
|
|||
return len( new_file_seeds )
|
||||
|
||||
|
||||
def IsURLFileSeeds( self ) -> bool:
|
||||
|
||||
if len( self._file_seeds ) == 0:
|
||||
|
||||
return True
|
||||
|
||||
|
||||
first = self._file_seeds[0]
|
||||
|
||||
return first.file_seed_type == FILE_SEED_TYPE_URL
|
||||
|
||||
|
||||
def NotifyFileSeedsUpdated( self, file_seeds: typing.Collection[ FileSeed ] ):
|
||||
|
||||
if len( file_seeds ) == 0:
|
||||
|
|
|
|||
|
|
@ -1,14 +1,14 @@
|
|||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusPSDHandling
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusBlurhash
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.images import HydrusImageMetadata
|
||||
from hydrus.core.images import HydrusImageOpening
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
from hydrus.core.files import HydrusPSDHandling
|
||||
from hydrus.core.files.images import HydrusBlurhash
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageMetadata
|
||||
from hydrus.core.files.images import HydrusImageOpening
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientFiles
|
||||
|
|
|
|||
|
|
@ -7,12 +7,12 @@ import typing
|
|||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusSerialisable
|
||||
from hydrus.core import HydrusThreading
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientData
|
||||
|
|
|
|||
|
|
@ -799,7 +799,7 @@ class SubscriptionQueryHeader( HydrusSerialisable.SerialisableBase ):
|
|||
file_seed_cache = query_log_container.GetFileSeedCache()
|
||||
|
||||
self._file_seed_cache_status = file_seed_cache.GetStatus()
|
||||
self._example_file_seed = file_seed_cache.GetExampleFileSeed()
|
||||
self._example_file_seed = file_seed_cache.GetExampleURLFileSeed()
|
||||
|
||||
|
||||
def WantsToResyncWithLogContainer( self ):
|
||||
|
|
|
|||
|
|
@ -6,9 +6,9 @@ from hydrus.core import HydrusConstants as HC
|
|||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPSDHandling
|
||||
from hydrus.core import HydrusSerialisable
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.files import HydrusPSDHandling
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientLocation
|
||||
|
|
|
|||
|
|
@ -24,13 +24,13 @@ from twisted.web.static import File as FileResource
|
|||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusTags
|
||||
from hydrus.core import HydrusTemp
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.networking import HydrusNetworkVariableHandling
|
||||
from hydrus.core.networking import HydrusServerRequest
|
||||
from hydrus.core.networking import HydrusServerResources
|
||||
|
|
@ -2505,6 +2505,8 @@ class HydrusResourceClientAPIRestrictedAddURLsGetURLFiles( HydrusResourceClientA
|
|||
|
||||
json_happy_url_statuses = []
|
||||
|
||||
we_only_saw_successful = True
|
||||
|
||||
for file_import_status in url_statuses:
|
||||
|
||||
if do_file_system_check:
|
||||
|
|
@ -2520,6 +2522,11 @@ class HydrusResourceClientAPIRestrictedAddURLsGetURLFiles( HydrusResourceClientA
|
|||
|
||||
json_happy_url_statuses.append( d )
|
||||
|
||||
if file_import_status.status not in CC.SUCCESSFUL_IMPORT_STATES:
|
||||
|
||||
we_only_saw_successful = False
|
||||
|
||||
|
||||
|
||||
body_dict = { 'normalised_url' : normalised_url, 'url_file_statuses' : json_happy_url_statuses }
|
||||
|
||||
|
|
@ -2527,6 +2534,12 @@ class HydrusResourceClientAPIRestrictedAddURLsGetURLFiles( HydrusResourceClientA
|
|||
|
||||
response_context = HydrusServerResources.ResponseContext( 200, mime = request.preferred_mime, body = body )
|
||||
|
||||
if we_only_saw_successful:
|
||||
|
||||
# not likely to change much, so no worries about reducing overhead here
|
||||
response_context.SetMaxAge( 30 )
|
||||
|
||||
|
||||
return response_context
|
||||
|
||||
|
||||
|
|
@ -2561,7 +2574,8 @@ class HydrusResourceClientAPIRestrictedAddURLsGetURLInfo( HydrusResourceClientAP
|
|||
|
||||
body = Dumps( body_dict, request.preferred_mime )
|
||||
|
||||
response_context = HydrusServerResources.ResponseContext( 200, mime = request.preferred_mime, body = body )
|
||||
# max age of ten minutes here
|
||||
response_context = HydrusServerResources.ResponseContext( 200, mime = request.preferred_mime, body = body, max_age = 600 )
|
||||
|
||||
return response_context
|
||||
|
||||
|
|
@ -3116,7 +3130,16 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
|
|||
'num_words' : file_info_manager.num_words,
|
||||
'has_audio' : file_info_manager.has_audio
|
||||
}
|
||||
|
||||
|
||||
filetype_forced = file_info_manager.FiletypeIsForced()
|
||||
|
||||
metadata_row[ 'filetype_forced' ] = filetype_forced
|
||||
|
||||
if filetype_forced:
|
||||
|
||||
metadata_row[ 'original_mime' ] = HC.mime_mimetype_string_lookup[ file_info_manager.original_mime ]
|
||||
|
||||
|
||||
if include_blurhash:
|
||||
|
||||
metadata_row[ 'blurhash' ] = file_info_manager.blurhash
|
||||
|
|
@ -3179,6 +3202,15 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
|
|||
'pixel_hash' : None if file_info_manager.pixel_hash is None else file_info_manager.pixel_hash.hex()
|
||||
}
|
||||
|
||||
filetype_forced = file_info_manager.FiletypeIsForced()
|
||||
|
||||
metadata_row[ 'filetype_forced' ] = filetype_forced
|
||||
|
||||
if filetype_forced:
|
||||
|
||||
metadata_row[ 'original_mime' ] = HC.mime_mimetype_string_lookup[ file_info_manager.original_mime ]
|
||||
|
||||
|
||||
if file_info_manager.mime in HC.MIMES_WITH_THUMBNAILS:
|
||||
|
||||
if width is not None and height is not None and width > 0 and height > 0:
|
||||
|
|
|
|||
|
|
@ -2381,7 +2381,7 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
|
|||
self._RecalculateMatchableSearchTexts()
|
||||
|
||||
|
||||
def ToString( self, with_count: bool = True, tag_display_type: int = ClientTags.TAG_DISPLAY_DISPLAY_ACTUAL, render_for_user: bool = False, or_under_construction: bool = False ) -> str:
|
||||
def ToString( self, with_count: bool = True, render_for_user: bool = False, or_under_construction: bool = False, for_parsable_export: bool = False ) -> str:
|
||||
|
||||
base = ''
|
||||
count_text = ''
|
||||
|
|
@ -2839,13 +2839,25 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
|
|||
is_phrase = 'is not'
|
||||
|
||||
|
||||
if len( hashes ) == 1:
|
||||
if len( hashes ) > 1:
|
||||
|
||||
base = '{} hash {} {}'.format( hash_type, is_phrase, hashes[0].hex() )
|
||||
is_phrase += ' in'
|
||||
|
||||
|
||||
if hash_type != 'sha256':
|
||||
|
||||
base = f'hash ({hash_type})'
|
||||
|
||||
|
||||
if len( hashes ) == 1 or for_parsable_export:
|
||||
|
||||
hashes_string = ', '.join( ( hash.hex() for hash in hashes ) )
|
||||
|
||||
base = f'{base} {is_phrase} {hashes_string}'
|
||||
|
||||
else:
|
||||
|
||||
base = '{} hash {} in {} hashes'.format( hash_type, is_phrase, HydrusData.ToHumanInt( len( hashes ) ) )
|
||||
base = f'{base} {is_phrase} {HydrusData.ToHumanInt( len( hashes ) )} hashes'
|
||||
|
||||
|
||||
|
||||
|
|
@ -2905,18 +2917,59 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
( hashes, max_hamming ) = self._value
|
||||
|
||||
base += ' {} files using max hamming of {}'.format( HydrusData.ToHumanInt( len( hashes ) ), max_hamming )
|
||||
if for_parsable_export:
|
||||
|
||||
hash_string = ', '.join( ( hash.hex() for hash in hashes ) )
|
||||
|
||||
else:
|
||||
|
||||
hash_string = f'{HydrusData.ToHumanInt( len( hashes ) )} files'
|
||||
|
||||
|
||||
base += f' {hash_string} with distance of {max_hamming}'
|
||||
|
||||
|
||||
elif self._predicate_type == PREDICATE_TYPE_SYSTEM_SIMILAR_TO_DATA:
|
||||
|
||||
base = 'similar to'
|
||||
base = 'similar to data'
|
||||
|
||||
if self._value is not None:
|
||||
|
||||
( pixel_hashes, perceptual_hashes, max_hamming ) = self._value
|
||||
|
||||
base += ' {} similar data hashes using max hamming of {}'.format( HydrusData.ToHumanInt( len( pixel_hashes ) + len( perceptual_hashes ) ), max_hamming )
|
||||
all_hashes = list( pixel_hashes ) + list( perceptual_hashes )
|
||||
|
||||
if for_parsable_export:
|
||||
|
||||
hash_string = ', '.join( ( hash.hex() for hash in all_hashes ) )
|
||||
|
||||
else:
|
||||
|
||||
components = []
|
||||
|
||||
if len( pixel_hashes ) > 0:
|
||||
|
||||
components.append( f'{HydrusData.ToHumanInt( len( pixel_hashes ) )} pixel')
|
||||
|
||||
|
||||
if len( perceptual_hashes ) > 0:
|
||||
|
||||
components.append( f'{HydrusData.ToHumanInt( len( perceptual_hashes ) )} perceptual')
|
||||
|
||||
|
||||
component_string = ', '.join( components )
|
||||
|
||||
hash_string = f'({component_string} hashes)'
|
||||
|
||||
|
||||
if len( perceptual_hashes ) > 0:
|
||||
|
||||
base += f' {hash_string} with distance of {max_hamming}'
|
||||
|
||||
else:
|
||||
|
||||
base += f' {hash_string}'
|
||||
|
||||
|
||||
|
||||
elif self._predicate_type == PREDICATE_TYPE_SYSTEM_FILE_SERVICE:
|
||||
|
|
|
|||
|
|
@ -105,8 +105,8 @@ options = {}
|
|||
# Misc
|
||||
|
||||
NETWORK_VERSION = 20
|
||||
SOFTWARE_VERSION = 556
|
||||
CLIENT_API_VERSION = 57
|
||||
SOFTWARE_VERSION = 557
|
||||
CLIENT_API_VERSION = 58
|
||||
|
||||
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )
|
||||
|
||||
|
|
|
|||
|
|
@ -739,6 +739,19 @@ class TagFilter( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
|
||||
|
||||
def SetRules( self, tag_slices, rule ):
|
||||
|
||||
with self._lock:
|
||||
|
||||
for tag_slice in tag_slices:
|
||||
|
||||
self._tag_slices_to_rules[ tag_slice ] = rule
|
||||
|
||||
|
||||
self._UpdateRuleCache()
|
||||
|
||||
|
||||
|
||||
def TagOK( self, tag, apply_unnamespaced_rules_to_namespaced_tags = False ):
|
||||
|
||||
with self._lock:
|
||||
|
|
|
|||
|
|
@ -7,8 +7,8 @@ from PIL import Image as PILImage
|
|||
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.images import HydrusImageOpening
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageOpening
|
||||
|
||||
def GetAnimationProperties( path, mime ):
|
||||
|
||||
|
|
@ -69,7 +69,7 @@ def GetZipAsPath( path_to_zip, path_in_zip="" ):
|
|||
return zipfile.Path( path_to_zip, at=path_in_zip )
|
||||
|
||||
|
||||
def IsOpenableZip( path_to_zip ):
|
||||
def IsEncryptedZip( path_to_zip ):
|
||||
|
||||
ENCRYPTED_FLAG = 0x1
|
||||
|
||||
|
|
@ -85,19 +85,49 @@ def IsOpenableZip( path_to_zip ):
|
|||
|
||||
if is_encrypted:
|
||||
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
except:
|
||||
|
||||
return False
|
||||
raise HydrusExceptions.DamagedOrUnusualFileException( 'Could not open this zip at all!' )
|
||||
|
||||
|
||||
|
||||
def filename_has_image_ext( filename: str ):
|
||||
|
||||
if '.' in filename:
|
||||
|
||||
ext_with_dot = '.' + filename.split( '.' )[-1]
|
||||
|
||||
if ext_with_dot in HC.IMAGE_FILE_EXTS:
|
||||
|
||||
return True
|
||||
|
||||
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def filename_has_video_ext( filename: str ):
|
||||
|
||||
if '.' in filename:
|
||||
|
||||
ext_with_dot = '.' + filename.split( '.' )[-1]
|
||||
|
||||
if ext_with_dot in HC.VIDEO_FILE_EXTS:
|
||||
|
||||
return True
|
||||
|
||||
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def ZipLooksLikeCBZ( path_to_zip ):
|
||||
|
||||
# TODO: we should probably wangle this away from 'zip' and towards 'archive', but it is fine as a first step
|
||||
|
|
@ -148,27 +178,22 @@ def ZipLooksLikeCBZ( path_to_zip ):
|
|||
continue
|
||||
|
||||
|
||||
if '.' in filename:
|
||||
if filename_has_image_ext( filename ):
|
||||
|
||||
ext_with_dot = '.' + filename.split( '.' )[-1]
|
||||
num_images += 1
|
||||
|
||||
if ext_with_dot in HC.IMAGE_FILE_EXTS:
|
||||
|
||||
num_images += 1
|
||||
|
||||
directories_to_image_filenames[ directory_path ].add( filename )
|
||||
|
||||
continue
|
||||
|
||||
elif ext_with_dot in HC.VIDEO_FILE_EXTS:
|
||||
|
||||
# this catches some zips nicely
|
||||
return False
|
||||
|
||||
else:
|
||||
|
||||
num_weird_files += 1
|
||||
|
||||
directories_to_image_filenames[ directory_path ].add( filename )
|
||||
|
||||
continue
|
||||
|
||||
elif filename_has_video_ext( filename ):
|
||||
|
||||
# this catches some zips nicely
|
||||
return False
|
||||
|
||||
else:
|
||||
|
||||
num_weird_files += 1
|
||||
|
||||
|
||||
|
||||
|
|
@ -191,16 +216,16 @@ def ZipLooksLikeCBZ( path_to_zip ):
|
|||
|
||||
directories_to_looks_good_scores = {}
|
||||
|
||||
for ( directory_path, filenames ) in directories_to_image_filenames.items():
|
||||
for ( directory_path, image_filenames ) in directories_to_image_filenames.items():
|
||||
|
||||
# ok, so a zip that has fifteen different filename styles is not a cbz
|
||||
# one that is all "Coolguy Adventures-c4-p001.jpg" however is!
|
||||
|
||||
# so let's take all the numbers and figure out how commonly the filenames are templated
|
||||
# so let's take all the numbers and figure out how commonly the image filenames are templated
|
||||
|
||||
unique_numberless_filenames = { re.sub( r'\d', '', filename ) for filename in filenames }
|
||||
unique_numberless_filenames = { re.sub( r'\d', '', filename ) for filename in image_filenames }
|
||||
|
||||
magical_uniqueness_percentage = len( unique_numberless_filenames ) / len( filenames )
|
||||
magical_uniqueness_percentage = ( len( unique_numberless_filenames ) - 1 ) / len( image_filenames )
|
||||
|
||||
directories_to_looks_good_scores[ directory_path ] = magical_uniqueness_percentage
|
||||
|
||||
|
|
@ -209,7 +234,7 @@ def ZipLooksLikeCBZ( path_to_zip ):
|
|||
|
||||
average_directory_good = sum( all_percentages ) / len( all_percentages )
|
||||
|
||||
# experimentally, I haven't seen it go above 0.138 on a legit cbz
|
||||
# experimentally, I haven't seen this go above 0.103 on a legit cbz
|
||||
if average_directory_good > 0.2:
|
||||
|
||||
return False
|
||||
|
|
@ -1,26 +1,26 @@
|
|||
import hashlib
|
||||
import os
|
||||
|
||||
from hydrus.core import HydrusAnimationHandling
|
||||
from hydrus.core import HydrusArchiveHandling
|
||||
from hydrus.core import HydrusClipHandling
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusDocumentHandling
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFlashHandling
|
||||
from hydrus.core import HydrusKritaHandling
|
||||
from hydrus.core import HydrusProcreateHandling
|
||||
from hydrus.core import HydrusPSDHandling
|
||||
from hydrus.core import HydrusPaths
|
||||
from hydrus.core import HydrusSerialisable
|
||||
from hydrus.core import HydrusSVGHandling
|
||||
from hydrus.core import HydrusPDFHandling
|
||||
from hydrus.core import HydrusTemp
|
||||
from hydrus.core import HydrusText
|
||||
from hydrus.core import HydrusUgoiraHandling
|
||||
from hydrus.core import HydrusVideoHandling
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusAnimationHandling
|
||||
from hydrus.core.files import HydrusArchiveHandling
|
||||
from hydrus.core.files import HydrusClipHandling
|
||||
from hydrus.core.files import HydrusDocumentHandling
|
||||
from hydrus.core.files import HydrusFlashHandling
|
||||
from hydrus.core.files import HydrusKritaHandling
|
||||
from hydrus.core.files import HydrusPDFHandling
|
||||
from hydrus.core.files import HydrusProcreateHandling
|
||||
from hydrus.core.files import HydrusPSDHandling
|
||||
from hydrus.core.files import HydrusSVGHandling
|
||||
from hydrus.core.files import HydrusUgoiraHandling
|
||||
from hydrus.core.files import HydrusVideoHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.networking import HydrusNetwork
|
||||
|
||||
try:
|
||||
|
|
@ -692,7 +692,14 @@ def GetMime( path, ok_to_look_for_hydrus_updates = False ):
|
|||
|
||||
if mime == HC.APPLICATION_ZIP:
|
||||
|
||||
if not HydrusArchiveHandling.IsOpenableZip( path ):
|
||||
try:
|
||||
|
||||
if HydrusArchiveHandling.IsEncryptedZip( path ):
|
||||
|
||||
return HC.APPLICATION_ZIP
|
||||
|
||||
|
||||
except HydrusExceptions.DamagedOrUnusualFileException:
|
||||
|
||||
return HC.APPLICATION_ZIP
|
||||
|
||||
|
|
@ -1,9 +1,9 @@
|
|||
import io
|
||||
import typing
|
||||
|
||||
from hydrus.core import HydrusArchiveHandling
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusArchiveHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from PIL import Image as PILImage
|
||||
import xml.etree.ElementTree as ET
|
||||
|
|
@ -4,11 +4,11 @@ import typing
|
|||
from PIL import Image as PILImage
|
||||
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
try:
|
||||
|
||||
from hydrus.core import HydrusPSDTools
|
||||
from hydrus.core.files import HydrusPSDTools
|
||||
|
||||
PSD_TOOLS_OK = True
|
||||
|
||||
|
|
@ -6,7 +6,7 @@ from psd_tools.api.numpy_io import has_transparency, get_transparency_index
|
|||
from psd_tools.api.pil_io import get_pil_mode, get_pil_channels, _create_image
|
||||
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core.images import HydrusImageNormalisation
|
||||
from hydrus.core.files.images import HydrusImageNormalisation
|
||||
|
||||
def PSDHasICCProfile( path: str ):
|
||||
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
from hydrus.core import HydrusArchiveHandling
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core.files import HydrusArchiveHandling
|
||||
import plistlib
|
||||
|
||||
# Mostly based on https://github.com/jaromvogel/ProcreateViewer/blob/master/ProcreatePython/ProcreateImageData.py
|
||||
|
|
@ -1,10 +1,10 @@
|
|||
import zipfile
|
||||
|
||||
from hydrus.core import HydrusArchiveHandling
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusTemp
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusArchiveHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
def ExtractFrame( path_to_zip, frame_index, extract_path ):
|
||||
|
||||
|
|
@ -58,7 +58,14 @@ def GetUgoiraProperties( path_to_zip ):
|
|||
|
||||
with zipfile.ZipFile( path_to_zip ) as zip_handle:
|
||||
|
||||
num_frames = len( zip_handle.infolist() )
|
||||
# let's discount the .js or .json
|
||||
|
||||
def is_js_stuff( s ):
|
||||
|
||||
return s.endswith( '.js' ) or s.endswith( '.json' )
|
||||
|
||||
|
||||
num_frames = len( [ 1 for zip_info in zip_handle.infolist() if not is_js_stuff( zip_info.filename ) ] )
|
||||
|
||||
|
||||
except:
|
||||
|
|
@ -5,12 +5,12 @@ import os
|
|||
import re
|
||||
import subprocess
|
||||
|
||||
from hydrus.core import HydrusAudioHandling
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusText
|
||||
from hydrus.core import HydrusThreading
|
||||
from hydrus.core.files import HydrusAudioHandling
|
||||
|
||||
FFMPEG_MISSING_ERROR_PUBBED = False
|
||||
FFMPEG_NO_CONTENT_ERROR_PUBBED = False
|
||||
|
|
@ -3,7 +3,7 @@ import cv2
|
|||
|
||||
from hydrus.external import blurhash as external_blurhash
|
||||
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
def GetBlurhashFromNumPy( numpy_image: numpy.array ) -> str:
|
||||
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
from hydrus.core.images import HydrusImageInit # right up top
|
||||
from hydrus.core.files.images import HydrusImageInit # right up top
|
||||
|
||||
import cv2
|
||||
import hashlib
|
||||
|
|
@ -29,12 +29,12 @@ from hydrus.core import HydrusConstants as HC
|
|||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusPSDHandling
|
||||
from hydrus.core import HydrusKritaHandling
|
||||
from hydrus.core.images import HydrusImageColours
|
||||
from hydrus.core.images import HydrusImageMetadata
|
||||
from hydrus.core.images import HydrusImageNormalisation
|
||||
from hydrus.core.images import HydrusImageOpening
|
||||
from hydrus.core.files import HydrusKritaHandling
|
||||
from hydrus.core.files import HydrusPSDHandling
|
||||
from hydrus.core.files.images import HydrusImageColours
|
||||
from hydrus.core.files.images import HydrusImageMetadata
|
||||
from hydrus.core.files.images import HydrusImageNormalisation
|
||||
from hydrus.core.files.images import HydrusImageOpening
|
||||
|
||||
def EnableLoadTruncatedImages():
|
||||
|
||||
|
|
@ -165,15 +165,6 @@ def GenerateNumPyImage( path, mime, force_pil = False ) -> numpy.array:
|
|||
|
||||
pil_image = HydrusImageOpening.RawOpenPILImage( path )
|
||||
|
||||
try:
|
||||
|
||||
pil_image.verify()
|
||||
|
||||
except Exception as e:
|
||||
|
||||
raise HydrusExceptions.UnsupportedFileException() from e
|
||||
|
||||
|
||||
if pil_image.mode == 'LAB':
|
||||
|
||||
force_pil = True
|
||||
|
|
@ -9,8 +9,8 @@ from PIL import ImageCms as PILImageCms
|
|||
|
||||
from hydrus.core import HydrusData
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core.images import HydrusImageColours
|
||||
from hydrus.core.images import HydrusImageMetadata
|
||||
from hydrus.core.files.images import HydrusImageColours
|
||||
from hydrus.core.files.images import HydrusImageMetadata
|
||||
|
||||
try:
|
||||
|
||||
|
|
@ -15,9 +15,9 @@ except:
|
|||
|
||||
from hydrus.core import HydrusConstants as HC
|
||||
from hydrus.core import HydrusExceptions
|
||||
from hydrus.core import HydrusFileHandling
|
||||
from hydrus.core import HydrusSerialisable
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files import HydrusFileHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.networking import HydrusNetwork
|
||||
|
||||
INT_PARAMS = { 'expires', 'num', 'since', 'content_type', 'action', 'status' }
|
||||
|
|
|
|||
|
|
@ -1300,19 +1300,45 @@ class ResponseContext( object ):
|
|||
return self._body_bytes
|
||||
|
||||
|
||||
def GetCookies( self ): return self._cookies
|
||||
def GetCookies( self ):
|
||||
|
||||
return self._cookies
|
||||
|
||||
|
||||
def GetMime( self ): return self._mime
|
||||
def GetMime( self ):
|
||||
|
||||
return self._mime
|
||||
|
||||
|
||||
def GetPath( self ): return self._path
|
||||
def GetPath( self ):
|
||||
|
||||
return self._path
|
||||
|
||||
|
||||
def GetStatusCode( self ): return self._status_code
|
||||
def GetStatusCode( self ):
|
||||
|
||||
return self._status_code
|
||||
|
||||
|
||||
def GetMaxAge( self ): return self._max_age
|
||||
def GetMaxAge( self ):
|
||||
|
||||
return self._max_age
|
||||
|
||||
|
||||
def HasBody( self ): return self._body_bytes is not None
|
||||
def SetMaxAge( self, age ):
|
||||
|
||||
self._max_age = age
|
||||
|
||||
|
||||
def HasPath( self ): return self._path is not None
|
||||
def HasBody( self ):
|
||||
|
||||
return self._body_bytes is not None
|
||||
|
||||
|
||||
def HasPath( self ):
|
||||
|
||||
return self._path is not None
|
||||
|
||||
|
||||
def IsAttachmentDownload( self ):
|
||||
|
||||
|
|
|
|||
|
|
@ -188,6 +188,7 @@ class Operators( Enum ):
|
|||
RELATIONAL_TIME = auto() # One of '=', '<', '>', UNICODE_APPROX_EQUAL ('≈') (takes '~=' too), and the various 'since', 'before', 'the day of', 'the month of' time-based analogues
|
||||
RELATIONAL_FOR_RATING_SERVICE = auto() # RELATIONAL, but in the middle of a 'service_name = 4/5' kind of thing
|
||||
EQUAL = auto() # One of '=' or '!='
|
||||
EQUAL_NOT_CONSUMING = auto() # One of '=' or '!=', doesn't consume this text so later things can look at it
|
||||
FILESERVICE_STATUS = auto() # One of 'is not currently in', 'is currently in', 'is not pending to', 'is pending to'
|
||||
TAG_RELATIONAL = auto() # A tuple of a string (a potential tag name) and a relational operator (as a string)
|
||||
ONLY_EQUAL = auto() # None (meaning =, since thats the only accepted operator)
|
||||
|
|
@ -241,7 +242,7 @@ SYSTEM_PREDICATES = {
|
|||
'similar to data': (Predicate.SIMILAR_TO_DATA, None, Value.SIMILAR_TO_HASHLIST_WITH_DISTANCE, None),
|
||||
'limit': (Predicate.LIMIT, Operators.ONLY_EQUAL, Value.NATURAL, None),
|
||||
'file ?type': (Predicate.FILETYPE, Operators.ONLY_EQUAL, Value.FILETYPE_LIST, None),
|
||||
'hash': (Predicate.HASH, Operators.EQUAL, Value.HASHLIST_WITH_ALGORITHM, None),
|
||||
'hash': (Predicate.HASH, Operators.EQUAL_NOT_CONSUMING, Value.HASHLIST_WITH_ALGORITHM, None),
|
||||
'archived? (date|time)|(date|time) archived|archived.': (Predicate.ARCHIVED_DATE, Operators.RELATIONAL_TIME, Value.DATE_OR_TIME_INTERVAL, None),
|
||||
'modified (date|time)|(date|time) modified|modified': (Predicate.MOD_DATE, Operators.RELATIONAL_TIME, Value.DATE_OR_TIME_INTERVAL, None),
|
||||
'last view(ed)? (date|time)|(date|time) last viewed|last viewed': (Predicate.LAST_VIEWED_TIME, Operators.RELATIONAL_TIME, Value.DATE_OR_TIME_INTERVAL, None),
|
||||
|
|
@ -406,28 +407,70 @@ def parse_value( string: str, spec ):
|
|||
|
||||
|
||||
elif spec == Value.SHA256_HASHLIST_WITH_DISTANCE:
|
||||
match = re.match( '(?P<hashes>([0-9a-f]+(\s|,)+)+)(with\s+)?distance\s+(?P<distance>0|([1-9][0-9]*))', string )
|
||||
match = re.match( '(?P<hashes>([0-9a-f]{4}[0-9a-f]+(\s|,)*)+)(with\s+)?(distance\s+)?(of\s+)?(?P<distance>0|([1-9][0-9]*))?', string )
|
||||
if match:
|
||||
hashes = set( hsh.strip() for hsh in re.sub( '\s', ' ', match[ 'hashes' ].replace( ',', ' ' ) ).split( ' ' ) if len( hsh ) > 0 )
|
||||
distance = int( match[ 'distance' ] )
|
||||
|
||||
d = match.groupdict()
|
||||
|
||||
if 'distance' in d and d[ 'distance' ] is not None:
|
||||
|
||||
distance = int( match[ 'distance' ] )
|
||||
|
||||
else:
|
||||
|
||||
distance = 4
|
||||
|
||||
|
||||
return string[ len( match[ 0 ] ): ], (hashes, distance)
|
||||
raise ValueError( "Invalid value, expected a list of hashes with distance" )
|
||||
elif spec == Value.SIMILAR_TO_HASHLIST_WITH_DISTANCE:
|
||||
match = re.match( '(?P<hashes>([0-9a-f]+(\s|,)+)+)(with\s+)?distance\s+(?P<distance>0|([1-9][0-9]*))', string )
|
||||
match = re.match( '(?P<hashes>([0-9a-f]{4}[0-9a-f]+(\s|,)*)+)(with\s+)?(distance\s+)?(of\s+)?(?P<distance>0|([1-9][0-9]*))?', string )
|
||||
if match:
|
||||
hashes = set( hsh.strip() for hsh in re.sub( '\s', ' ', match[ 'hashes' ].replace( ',', ' ' ) ).split( ' ' ) if len( hsh ) > 0 )
|
||||
pixel_hashes = { hash for hash in hashes if len( hash ) == 64 }
|
||||
perceptual_hashes = { hash for hash in hashes if len( hash ) == 16 }
|
||||
distance = int( match[ 'distance' ] )
|
||||
|
||||
d = match.groupdict()
|
||||
|
||||
if 'distance' in d and d[ 'distance' ] is not None:
|
||||
|
||||
distance = int( match[ 'distance' ] )
|
||||
|
||||
else:
|
||||
|
||||
distance = 8
|
||||
|
||||
|
||||
return string[ len( match[ 0 ] ): ], (pixel_hashes, perceptual_hashes, distance)
|
||||
raise ValueError( "Invalid value, expected a list of hashes with distance" )
|
||||
elif spec == Value.HASHLIST_WITH_ALGORITHM:
|
||||
match = re.match( '(?P<hashes>([0-9a-f]+(\s|,)*)+)((with\s+)?algorithm)?\s*(?P<algorithm>sha256|sha512|md5|sha1|)', string )
|
||||
|
||||
# hydev KISS hijack here, instead of clever regex to capture algorithm in all sorts of situations, let's just grab the hex we see and scan the rest for non-hex phrases mate
|
||||
# old pattern: match = re.match( '(?P<hashes>([0-9a-f]+(\s|,)*)+)((with\s+)?algorithm)?\s*(?P<algorithm>sha256|sha512|md5|sha1|)', string )
|
||||
|
||||
algorithm = 'sha256'
|
||||
|
||||
for possible_algorithm in ( 'md5', 'sha1', 'sha512' ):
|
||||
|
||||
if possible_algorithm in string:
|
||||
|
||||
algorithm = possible_algorithm
|
||||
|
||||
break
|
||||
|
||||
|
||||
|
||||
# {8} here to make sure we are looking at proper hash hex and not some short 'a' or 'de' word
|
||||
match = re.search( '(?P<hashes>([0-9a-f]{8}[0-9a-f]+(\s|,)*)+)', string )
|
||||
|
||||
if match:
|
||||
hashes = set( hsh.strip() for hsh in re.sub( '\s', ' ', match[ 'hashes' ].replace( ',', ' ' ) ).split( ' ' ) if len( hsh ) > 0 )
|
||||
algorithm = match[ 'algorithm' ] if len( match[ 'algorithm' ] ) > 0 else 'sha256'
|
||||
return string[ len( match[ 0 ] ): ], (hashes, algorithm)
|
||||
raise ValueError( "Invalid value, expected a list of hashes with algorithm" )
|
||||
return string[ match.endpos : ], (hashes, algorithm)
|
||||
|
||||
raise ValueError( "Invalid value, expected a list of hashes and perhaps an algorithm" )
|
||||
|
||||
|
||||
elif spec == Value.FILETYPE_LIST:
|
||||
valid_values = sorted( FILETYPES.keys(), key = lambda k: len( k ), reverse = True )
|
||||
ftype_regex = '(' + '|'.join( [ '(' + val + ')' for val in valid_values ] ) + ')'
|
||||
|
|
@ -740,12 +783,26 @@ def parse_operator( string: str, spec ):
|
|||
raise ValueError( "Invalid rating operator" )
|
||||
elif spec == Operators.EQUAL:
|
||||
if string.startswith( '==' ): return string[ 2: ], '='
|
||||
if string.startswith( '=' ): return string[ 1: ], '='
|
||||
if string.startswith( UNICODE_NOT_EQUAL ): return string[ 1: ], '!='
|
||||
if string.startswith( '!=' ): return string[ 2: ], '!='
|
||||
if string.startswith( '=' ): return string[ 1: ], '='
|
||||
if string.startswith( 'is not' ): return string[ 6: ], '!='
|
||||
if string.startswith( 'is' ): return string[ 2: ], '='
|
||||
if string.startswith( 'isn\'t' ): return string[ 5: ], '!='
|
||||
if string.startswith( 'is' ): return string[ 2: ], '='
|
||||
raise ValueError( "Invalid equality operator" )
|
||||
elif spec == Operators.EQUAL_NOT_CONSUMING:
|
||||
|
||||
# hydev checking in here with some nonsense that catches an awkward situation
|
||||
# system:hash (md5) = blah
|
||||
# we want to see the = but not eat the md5, so in this special case, which isn't hard to parse otherwise, we'll just look for it and return no changes
|
||||
|
||||
if '==' in string: return string, '='
|
||||
if UNICODE_NOT_EQUAL in string: return string, '!='
|
||||
if '!=' in string: return string, '!='
|
||||
if '=' in string: return string, '='
|
||||
if 'is not' in string: return string, '!='
|
||||
if 'isn\'t' in string: return string, '!='
|
||||
if 'is' in string: return string, '='
|
||||
raise ValueError( "Invalid equality operator" )
|
||||
elif spec == Operators.FILESERVICE_STATUS:
|
||||
match = re.match( '(is )?currently in', string )
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ from hydrus.core import HydrusGlobals as HG
|
|||
from hydrus.core import HydrusTags
|
||||
from hydrus.core import HydrusText
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
from hydrus.client import ClientAPI
|
||||
|
|
@ -4884,6 +4884,11 @@ class TestClientAPI( unittest.TestCase ):
|
|||
ratings_dict[ HG.test_controller.example_incdec_rating_service_key ] = int( random.random() * 16 )
|
||||
|
||||
|
||||
if random.random() > 0.8:
|
||||
|
||||
file_info_manager.original_mime = HC.IMAGE_PNG
|
||||
|
||||
|
||||
ratings_manager = ClientMediaManagers.RatingsManager( {} )
|
||||
notes_manager = ClientMediaManagers.NotesManager( { 'note' : 'hello', 'note2' : 'hello2' } )
|
||||
file_viewing_stats_manager = ClientMediaManagers.FileViewingStatsManager.STATICGenerateEmptyManager( timestamps_manager )
|
||||
|
|
@ -4921,6 +4926,15 @@ class TestClientAPI( unittest.TestCase ):
|
|||
'num_words' : file_info_manager.num_words
|
||||
}
|
||||
|
||||
filetype_forced = file_info_manager.FiletypeIsForced()
|
||||
|
||||
metadata_row[ 'filetype_forced' ] = filetype_forced
|
||||
|
||||
if filetype_forced:
|
||||
|
||||
metadata_row[ 'original_mime' ] = HC.mime_mimetype_string_lookup[ file_info_manager.original_mime ]
|
||||
|
||||
|
||||
only_return_basic_information_metadata.append( dict( metadata_row ) )
|
||||
|
||||
metadata_row[ 'blurhash' ] = file_info_manager.blurhash
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ from hydrus.core import HydrusData
|
|||
from hydrus.core import HydrusGlobals as HG
|
||||
from hydrus.core import HydrusSerialisable
|
||||
from hydrus.core import HydrusTime
|
||||
from hydrus.core.images import HydrusImageHandling
|
||||
from hydrus.core.files.images import HydrusImageHandling
|
||||
from hydrus.core.networking import HydrusNetwork
|
||||
|
||||
from hydrus.client import ClientConstants as CC
|
||||
|
|
|
|||
|
|
@ -1843,7 +1843,7 @@ class TestTagObjects( unittest.TestCase ):
|
|||
|
||||
p = ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_SYSTEM_HASH, ( ( bytes.fromhex( 'abcd' ), ), 'sha256' ) )
|
||||
|
||||
self.assertEqual( p.ToString(), 'system:sha256 hash is abcd' )
|
||||
self.assertEqual( p.ToString(), 'system:hash is abcd' )
|
||||
self.assertEqual( p.GetNamespace(), 'system' )
|
||||
self.assertEqual( p.GetTextsAndNamespaces( render_for_user ), [ ( p.ToString(), 'namespace', p.GetNamespace() ) ] )
|
||||
|
||||
|
|
@ -1941,13 +1941,13 @@ class TestTagObjects( unittest.TestCase ):
|
|||
|
||||
p = ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_SYSTEM_SIMILAR_TO_FILES, ( ( bytes.fromhex( 'abcd' ), ), 5 ) )
|
||||
|
||||
self.assertEqual( p.ToString(), 'system:similar to 1 files using max hamming of 5' )
|
||||
self.assertEqual( p.ToString(), 'system:similar to 1 files with distance of 5' )
|
||||
self.assertEqual( p.GetNamespace(), 'system' )
|
||||
self.assertEqual( p.GetTextsAndNamespaces( render_for_user ), [ ( p.ToString(), 'namespace', p.GetNamespace() ) ] )
|
||||
|
||||
p = ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_SYSTEM_SIMILAR_TO_DATA, ( ( os.urandom( 32 ), ), ( os.urandom( 32 ), ), 2 ) )
|
||||
|
||||
self.assertEqual( p.ToString(), 'system:similar to 2 similar data hashes using max hamming of 2' )
|
||||
self.assertEqual( p.ToString(), 'system:similar to data (1 pixel, 1 perceptual hashes) with distance of 2' )
|
||||
self.assertEqual( p.GetNamespace(), 'system' )
|
||||
self.assertEqual( p.GetTextsAndNamespaces( render_for_user ), [ ( p.ToString(), 'namespace', p.GetNamespace() ) ] )
|
||||
|
||||
|
|
@ -2021,11 +2021,15 @@ class TestTagObjects( unittest.TestCase ):
|
|||
def test_system_predicate_parsing( self ):
|
||||
|
||||
for ( expected_result_text, sys_pred_text ) in [
|
||||
( 'system:similar to 4 files using max hamming of 3', "system:similar to e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c08 e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c09 e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c10, e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c11 with distance 3" ),
|
||||
( 'system:similar to 1 files using max hamming of 5', "system:similar to e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c00 distance 5" ),
|
||||
( 'system:similar to 2 similar data hashes using max hamming of 4', "system:similar to data b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd91 b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd92 distance 4" ),
|
||||
( 'system:similar to 2 similar data hashes using max hamming of 4', "system:similar to data 0702790ffeae5c8a 51ad07c228ab7469 distance 4" ),
|
||||
( 'system:similar to 4 similar data hashes using max hamming of 4', "system:similar to data b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd91 b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd92 51ad07c228ab7469 0702790ffeae5c8a distance 4" ),
|
||||
( 'system:similar to 4 files with distance of 3', "system:similar to e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c08 e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c09 e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c10, e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c11 with distance 3" ),
|
||||
( 'system:similar to 1 files with distance of 5', "system:similar to e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c00 distance 5" ),
|
||||
( 'system:similar to 1 files with distance of 4', "system:similar to e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c00" ),
|
||||
( 'system:similar to 1 files with distance of 5', "system:similar to e2c1592ce2a3338767bb7990738ae06357cbdfe917669da9a0d04069f9759c00 with distance of 5" ),
|
||||
( 'system:similar to data (2 pixel hashes)', "system:similar to data b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd91 b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd92 distance 4" ),
|
||||
( 'system:similar to data (2 pixel hashes)', "system:similar to data b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd91 b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd92" ),
|
||||
( 'system:similar to data (2 perceptual hashes) with distance of 4', "system:similar to data 0702790ffeae5c8a 51ad07c228ab7469 distance 4" ),
|
||||
( 'system:similar to data (2 perceptual hashes) with distance of 8', "system:similar to data 0702790ffeae5c8a 51ad07c228ab7469" ),
|
||||
( 'system:similar to data (2 pixel, 2 perceptual hashes) with distance of 4', "system:similar to data b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd91 b51d75120456d6e2155416f26416a96290b0a524bf1582af50b4fcf46dedcd92 51ad07c228ab7469 0702790ffeae5c8a distance 4" ),
|
||||
( 'system:everything', "system:everything" ),
|
||||
( 'system:inbox', "system:inbox " ),
|
||||
( 'system:archive', "system:archive " ),
|
||||
|
|
@ -2071,13 +2075,13 @@ class TestTagObjects( unittest.TestCase ):
|
|||
( 'system:filetype is apng, jpeg, png', "system:filetype = image/jpg, image/png, apng" ),
|
||||
( 'system:filetype is image', "system:filetype is image" ),
|
||||
( 'system:filetype is animated gif, static gif, jpeg', "system:filetype = static gif, animated gif, jpeg" ),
|
||||
( 'system:sha256 hash is in 3 hashes', "system:hash = abcdef01 abcdef02 abcdef03" ),
|
||||
( 'system:md5 hash is in 3 hashes', "system:hash = abcdef01 abcdef, abcdef04 md5" ),
|
||||
( 'system:md5 hash is abcdef01', "system:hash = abcdef01 md5" ),
|
||||
( 'system:md5 hash is abcdef01', "system:Hash = Abcdef01 md5" ),
|
||||
( 'system:md5 hash is not abcdef01', "system:Hash != Abcdef01 md5" ),
|
||||
( 'system:md5 hash is not abcdef01', "system:Hash is not Abcdef01 md5" ),
|
||||
( 'system:sha256 hash is abcdef0102', "system:hash = abcdef0102" ),
|
||||
( 'system:hash is in 3 hashes', "system:hash = cf09faad262075f96bf9a30052b8ec224e096948a4f3a2776df5fa5a777bcfd8 a1b0ab771d11d9a6d1f993efee9d253d3aa78914387a7c8ceab520af88ab3de2 98a7d2f4735a5fcc70e7c94e2dadcc6ea45123fb2035b9cfe7ad1d78e48cae9e" ),
|
||||
( 'system:hash (md5) is in 3 hashes', "system:hash = ada7a31713ba24652c52e52c6f212e47 546fd4b8c39fc53e77e2f28b59cd1b18, cec888bacb79825621738454a4c9d226 md5" ),
|
||||
( 'system:hash (md5) is 666d0a395c8d4eebb5b15a0771972a01', "system:hash (md5) = 666d0a395c8d4eebb5b15a0771972a01" ),
|
||||
( 'system:hash (md5) is 123fec741ebe7596c1faf8d7689693b8', "system:Hash = 123feC741ebe7596c1faf8d7689693b8 md5" ),
|
||||
( 'system:hash (sha1) is not 2496baf1ded134b5ff7e44f72155240b9561ab5a', "system:Hash (sha1) != 2496baf1ded134b5ff7e44f72155240b9561ab5a" ),
|
||||
( 'system:hash is not b49f25453c6351403d62cc4d065321106c90f98b5653e83d289dbe0d55ba8c94', "system:Hash is not b49f25453c6351403d62cc4d065321106c90f98b5653e83d289dbe0d55ba8c94 sha256" ),
|
||||
( 'system:hash is 4d4ff3d42459f824295a36138782c444028f533b6ae5f0f67b27e9bf3c93de5d', "system:hash = 4d4ff3d42459f824295a36138782c444028f533b6ae5f0f67b27e9bf3c93de5d" ),
|
||||
( 'system:archived time: since 60 days ago', "system:archived date < 60 days" ),
|
||||
( 'system:archived time: since 60 days ago', "system:archive date < 60 days" ),
|
||||
( 'system:archived time: since 60 days ago', "system:archived time < 60 days" ),
|
||||
|
|
|
|||
Loading…
Reference in New Issue