RepoMirrors
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Version 550 

closes #1444, closes #1447, closes #1458
This commit is contained in:
Hydrus Network Developer 2023-11-01 16:38:03 -05:00
parent bd1348287d
commit 1f9b4ba892
No known key found for this signature in database
GPG Key ID: 76249F053212133C
41 changed files with 586 additions and 720 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

66
docs/changelog.md
View File

@ -7,6 +7,40 @@ title: Changelog
!!! note
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
## [Version 550](https://github.com/hydrusnetwork/hydrus/releases/tag/v550)
### misc
* if you enter invalid URLs (i.e. non-parsing) into 'manage URLs', the dialog now lets you know they were not apparently good and asks if you want to enter them anyway. previously, it errored-out and disallowed anything that wasn't parsing ok (issue #1444)
* when physically deleting files (i.e. deleting from trash or picking 'permanently delete' from the advanced delete dialog), the relevant files are now immediately removed from view. there were some situations where, when physically deleting a lot of files (causing the job to clear in batches), you could subsequently click on a soon-to-be-deleted file, loading it in mpv, and then, if you started a big UI-lag job like loading 'manage siblings', it could cause a crash if the file was deleted during the UI hang (issue #1447)
* the client now explicitly closes and clears its network connections after five minutes of inactivity. it turns out that the behind the scenes tools were not doing this exactly as I had thought, clogging up connection slots (issue #1458)
* thanks to a user, the rendering of palettized PNGs with ICC profiles is fixed!
* fixed the github build script to include the new-as-of-a-couple-of-weeks-ago 'auto_update_installer.bat' file in the Windows builds. sorry for the confusion here, I forgot I had to do this!
* optimised deselection of a large number of files when you already have a lot of thumbnails selected (a tricky example of this is clicking on an unselected file when you have a lot of files selected, thus deselecting all that old stuff). should be a little faster to work on big lists now
* further optimised reduction recalculation of the taglist in general
### thumbnail fill
* after vacillating and talking about it for months, I finally reworked how ''scale to fill' thumbnails work. as sometimes happens, I only had to change about six critical lines of code to get the core functionality changed and nothing seems to have exploded
* the main change here is KISS--'fill' thumbnail image files on disk are no longer clipped to just the viewable area, but the whole image scaled to fill the thumbnail space (with exceptions for extreme cases). this change gives us some simplicity and flexibility behind the scenes, saves some regeneration work when the user only changes one thumbnail dimension setting, improves maintenance tasks based off the thumbnail (like blurhash), and means that the Client API can fetch your thumbs and still have something useful to display
* if you have 'scale to fit' set, hydrus will regenerate your thumbnails naturally as you browse the client. fingers crossed, you won't notice any visual difference through the transition
* 'open externally' button panels now display their thumbnails with more reasonable maximum dimensions, and when things are gonk for whatever reason, they should nonetheless be centered correctly
* as a side thing, this change allowed me to finally purge all the clipping tech from the thumbnail pipeline, where it had obtusely sunk in to every possible filetype thumbgen
### eager login system
* I fixed a problem where some sorts of login script could allow a network job supposedly waiting on them to start before they had completed. it was due to a complicated 'am I logged in?' cookie testing issue while the login process was still working. all network jobs that hypothetically need a login now test if there is a login process currently working on their domain and will properly wait for that process to finish before they move on
* fixed a 'cannot log in' reporting bug in the login system
* some misc login code cleanup
### smarter orphan file record and repository update handling
* _this is advanced stuff, most users can ignore_
* _database->db maintenance->clear orphan file records_ is now able to recover file records where A) the file is in a service component but not the master, B) the file exists on disk. it copies the import timestamp from the specific to the umbrella domain and spams all the repaired files to a new page for user review. this maintenance routine isn't used all that much, but when you have a damaged database, it is nice to recover as much as possible rather than having to export (with clear orphan file records+clear orphan files) and then reimport and lose archive/inbox status and import timestamps
* repository update files now have a 'delete from repository updates' entry in their right-click menu
* this area of the code appears to be related to the PTR 404 issue some users have had (it seems to be repository update records not beeing added/deleted/updated correctly), so I am likely to revisit this
* deleting a file from 'all local files' (which happens for repository update files) now correctly updates the UI-level media object to recognise that the file is fully deleted from all local file domains beneath the umbrella, removing the 'delete from x' commands from their menu, and in the right view contexts removing them from view completely
## [Version 549](https://github.com/hydrusnetwork/hydrus/releases/tag/v549)
### misc
@ -358,35 +392,3 @@ title: Changelog
* the feature to migrate the SQLite database files and then restart is removed from the 'migrate database' dialog. it was always ultrajank in a place that really shouldn't be, and it was completely user-unfriendly. just move things manually, while the client is closed
* the old 'recover and merge surplus database locations into the correct position' side feature in 'move files now' is removed. it was always a little jank, was very rarely actually helpful, and had zero reporting. it will return in the new system as a better one-shot maintenance job
* touched up the migrated database help a little
## [Version 540](https://github.com/hydrusnetwork/hydrus/releases/tag/v540)
### misc
* the system predicate parser can now handle 'system:filetype is xxx' for more of the general human-friendly filetype strings like 'video' and 'mkv'. it can also handle 'static gif' and any other types with spaces but now enforces commas between each filetype. I think all system:filetype predicate strings the client produces now parse correctly if you paste them back
* fixed many bitmap imports, most typically in the 'system:similar files' system, which was not generating pixel hashes correctly. most/all bitmaps coming in with alpha channels, or, I also believe, with a null channel (RGB32), were being handled wrong and coming out BGR. perceptual hashes are greyscale and were not affected, but pixel hashes were wrong. this was a real pain to figure out, and it may be that it is still broken for users on big-endian systems or something, so let me know how you get on
* added links to https://github.com/abtalerico/wd-hydrus-tagger (danbooru-trained model tagging) and https://github.com/Garbevoir/wd-e621-hydrus-tagger (which adds more models) to the client api help. reports are that they work well, even on 'normal' pictures
* the bad darkmode tooltip text colour in the new Qt 6.5.2 on Windows appears to be a bug, here: https://bugreports.qt.io/browse/QTBUG-116021 . there's not a great answer here, so let me know your thoughts. if you like, you can edit a custom stylesheet with a different `QToolTip` `background-color`, or I can spam some alternate fixed QSS files for everyone, or we can wait for a fix on Qt's end
* on update, all existing PSD and static gif files will be scheduled for pixel hash regen, perceptual hash regen, and entry into the similar files system (I forgot to do this last week)
* on update, all existing PNGs will be scheduled for pixel duplicate data regen. we have a legacy alpha channel issue here that has reared its head several times (searchting for 'must not be pixel dupes', but getting pixel dupes), so I am just going to bosh it on the head for everyone
### file maintenance
* if a file has multiple jobs pending, the file maintenance manager now processes all those jobs at once, saving significant disk I/O. also, a couple things like the 'do all work' button's popup now shows the total number of jobs to do, rather than that of each job type in turn
* the 'manage scheduled jobs' file maintenance panel now shows the count for jobs that exist but are not yet due. previously, these were hidden, which was part of the mkv/webm duplicate difficulties last week.
* when the program needs to rename a file because it has a new mime, it now first tests if the file is still in use (normally this means some file parsing component like ffmpeg or opencv is still cleaning up OS file handles or whatever), and, if so, waits just a little bit before trying
* relatedly, the 'try and delete the rename-dupe again' job now tries again in one hour, rather than one week in the future, and if that after-one-hour job fails again (this would usually be because you were actually viewing the original file in the media viewer at the time of its reparse), then that job will retry again in a week, and the week after if that fails, and again after that, etc... for about a year
* fixed an issue with the thumbnail resizing maintenance job on PSD files and probably some other weird types too
* fixed some scheduling issues in how the mainloop of the file maintenance system tests its current rate of work and when it should cancel a current batch of work
### boring stuff
* simplified and cleaned up some of the duplicate system king-fetching code. I _may_ have also fixed one instance of pair representatives being fetched wrong for the filter when 'at least one file has to match the one search'
* when editing the duplicate action merge options, a new label at the top says which dupe action you are editing for, and if it isn't "this is better", it notes that the available merge actions are limited
* improved four things with the recovery code that handles missing master hash definitions--first, the substitute hashes are now the correct length; second, they are now saved back to the database, which should stop issues like the "trying to delete a thing that doesn't exist and has an ever-changing name in a loop forever" bug; third, the popup tells the user what to do next, and more information is written to the log; and fourth, the client checks the local hash cache so see if it can automatically recover the missing data
### client api
* the `file_metadata` call now has two new fields, `filetype_human`, which looks like 'jpeg' or 'webm', and `filetype_enum`, which uses internal hydrus filetype numbers
* the help and unit tests are updated for this
* client api version is now 50

10
docs/developer_api.md
View File

@ -1872,10 +1872,14 @@ Response:
If hydrus keeps no thumbnail for the filetype, for instance with pdfs, then you will get the same default 'pdf' icon you see in the client. If the file does not exist in the client, or the thumbnail was expected but is missing from storage, you will get the fallback 'hydrus' icon, again just as you would in the client itself. This request should never give a 404.
!!! note
If you get a 'default' filetype thumbnail like the pdf or hydrus one, you will be pulling the defaults straight from the hydrus/static folder. They will most likely be 200x200 pixels.
!!! note "Size of Normal Thumbs"
Thumbnails are not guaranteed to be the correct size! If a thumbnail has not been loaded in the client in years, it could well have been fitted for older thumbnail settings. Also, even 'clean' thumbnails will not always fit inside the settings' bounding box; they may be boosted due to a high-DPI setting or spill over due to a 'fill' vs 'fit' preference. You cannot easily predict what resolution a thumbnail will or should have!
In general, thumbnails *are* the correct ratio. If you are drawing thumbs, you should embed them to fit or fill, but don't fix them at 100% true size: make sure they can scale to the size you want!
!!! note "Size of Defaults"
If you get a 'default' filetype thumbnail like the pdf or hydrus one, you will be pulling the pngs straight from the hydrus/static folder. They will most likely be 200x200 pixels.
### **GET `/get_files/render`** { id="get_files_render" }
_Get an image file as rendered by Hydrus._

2
docs/getting_started_subscriptions.md
View File

@ -52,7 +52,7 @@ _(you might like to come back to this point once you have tried subs for a week
## ok, I set up three hundred queries, and now these popup buttons are a hassle { id="presentation" }
One the edit subscription panel, the 'presentation' options let you publish files to a page. The page will have the subscription's name, just like the button makes, but it cuts out the middle-man and 'locks it in' more than the button, which will be forgotten if you restart the client. **Also, if a page with that name already exists, the new files will be appended to it, just like a normal import page!** I strongly recommend moving to this once you have several subs going. Make a 'page of pages' called 'subs' and put all your subscription landing pages in there, and then you can check it whenever is convenient.
On the edit subscription panel, the 'presentation' options let you publish files to a page. The page will have the subscription's name, just like the button makes, but it cuts out the middle-man and 'locks it in' more than the button, which will be forgotten if you restart the client. **Also, if a page with that name already exists, the new files will be appended to it, just like a normal import page!** I strongly recommend moving to this once you have several subs going. Make a 'page of pages' called 'subs' and put all your subscription landing pages in there, and then you can check it whenever is convenient.
If you discover your subscription workflow tends to be the same for each sub, you can also customise the publication 'label' used. If multiple subs all publish to the 'nsfw subs' label, they will all end up on the same 'nsfw subs' popup button or landing page. Sending multiple subscriptions' import streams into just one or two locations like this can be great.

29
docs/old_changelog.html
View File

@ -34,6 +34,35 @@
<div class="content">
<h1 id="changelog"><a href="#changelog">changelog</a></h1>
<ul>
<li>
<h2 id="version_550"><a href="#version_550">version 550</a></h2>
<ul>
<li><h3>misc</h3></li>
<li>if you enter invalid URLs (i.e. non-parsing) into 'manage URLs', the dialog now lets you know they were not apparently good and asks if you want to enter them anyway. previously, it errored-out and disallowed anything that wasn't parsing ok (issue #1444)</li>
<li>when physically deleting files (i.e. deleting from trash or picking 'permanently delete' from the advanced delete dialog), the relevant files are now immediately removed from view. there were some situations where, when physically deleting a lot of files (causing the job to clear in batches), you could subsequently click on a soon-to-be-deleted file, loading it in mpv, and then, if you started a big UI-lag job like loading 'manage siblings', it could cause a crash if the file was deleted during the UI hang (issue #1447)</li>
<li>the client now explicitly closes and clears its network connections after five minutes of inactivity. it turns out that the behind the scenes tools were not doing this exactly as I had thought, clogging up connection slots (issue #1458)</li>
<li>thanks to a user, the rendering of palettized PNGs with ICC profiles is fixed!</li>
<li>fixed the github build script to include the new-as-of-a-couple-of-weeks-ago 'auto_update_installer.bat' file in the Windows builds. sorry for the confusion here, I forgot I had to do this!</li>
<li>optimised deselection of a large number of files when you already have a lot of thumbnails selected (a tricky example of this is clicking on an unselected file when you have a lot of files selected, thus deselecting all that old stuff). should be a little faster to work on big lists now</li>
<li>further optimised reduction recalculation of the taglist in general</li>
<li><h3>thumbnail fill</h3></li>
<li>after vacillating and talking about it for months, I finally reworked how ''scale to fill' thumbnails work. as sometimes happens, I only had to change about six critical lines of code to get the core functionality changed and nothing seems to have exploded</li>
<li>the main change here is KISS--'fill' thumbnail image files on disk are no longer clipped to just the viewable area, but the whole image scaled to fill the thumbnail space (with exceptions for extreme cases). this change gives us some simplicity and flexibility behind the scenes, saves some regeneration work when the user only changes one thumbnail dimension setting, improves maintenance tasks based off the thumbnail (like blurhash), and means that the Client API can fetch your thumbs and still have something useful to display</li>
<li>if you have 'scale to fit' set, hydrus will regenerate your thumbnails naturally as you browse the client. fingers crossed, you won't notice any visual difference through the transition</li>
<li>'open externally' button panels now display their thumbnails with more reasonable maximum dimensions, and when things are gonk for whatever reason, they should nonetheless be centered correctly</li>
<li>as a side thing, this change allowed me to finally purge all the clipping tech from the thumbnail pipeline, where it had obtusely sunk in to every possible filetype thumbgen</li>
<li><h3>eager login system</h3></li>
<li>I fixed a problem where some sorts of login script could allow a network job supposedly waiting on them to start before they had completed. it was due to a complicated 'am I logged in?' cookie testing issue while the login process was still working. all network jobs that hypothetically need a login now test if there is a login process currently working on their domain and will properly wait for that process to finish before they move on</li>
<li>fixed a 'cannot log in' reporting bug in the login system</li>
<li>some misc login code cleanup</li>
<li><h3>smarter orphan file record and repository update handling</h3></li>
<li>_this is advanced stuff, most users can ignore_</li>
<li>_database->db maintenance->clear orphan file records_ is now able to recover file records where A) the file is in a service component but not the master, B) the file exists on disk. it copies the import timestamp from the specific to the umbrella domain and spams all the repaired files to a new page for user review. this maintenance routine isn't used all that much, but when you have a damaged database, it is nice to recover as much as possible rather than having to export (with clear orphan file records+clear orphan files) and then reimport and lose archive/inbox status and import timestamps</li>
<li>repository update files now have a 'delete from repository updates' entry in their right-click menu</li>
<li>this area of the code appears to be related to the PTR 404 issue some users have had (it seems to be repository update records not beeing added/deleted/updated correctly), so I am likely to revisit this</li>
<li>deleting a file from 'all local files' (which happens for repository update files) now correctly updates the UI-level media object to recognise that the file is fully deleted from all local file domains beneath the umbrella, removing the 'delete from x' commands from their menu, and in the right view contexts removing them from view completely</li>
</ul>
</li>
<li>
<h2 id="version_549"><a href="#version_549">version 549</a></h2>
<ul>

18
hydrus/client/ClientFiles.py
View File

@ -597,13 +597,13 @@ class ClientFilesManager( object ):
thumbnail_scale_type = self._controller.new_options.GetInteger( 'thumbnail_scale_type' )
thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
( clip_rect, target_resolution ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( ( width, height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
target_resolution = HydrusImageHandling.GetThumbnailResolution( ( width, height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
percentage_in = self._controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )
try:
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( file_path, target_resolution, mime, duration, num_frames, clip_rect = clip_rect, percentage_in = percentage_in )
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( file_path, target_resolution, mime, duration, num_frames, percentage_in = percentage_in )
except Exception as e:
@ -1615,6 +1615,18 @@ class ClientFilesManager( object ):
return path
def LocklessHasFile( self, hash, mime ):
path = self._GenerateExpectedFilePath( hash, mime )
if HG.file_report_mode:
HydrusData.ShowText( 'File path test: ' + path )
return os.path.exists( path )
def LocklessHasThumbnail( self, hash ):
path = self._GenerateExpectedThumbnailPath( hash )
@ -1743,7 +1755,7 @@ class ClientFilesManager( object ):
thumbnail_scale_type = self._controller.new_options.GetInteger( 'thumbnail_scale_type' )
thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
( clip_rect, ( expected_width, expected_height ) ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( (media_width, media_height), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
( expected_width, expected_height ) = HydrusImageHandling.GetThumbnailResolution( (media_width, media_height), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
if current_width != expected_width or current_height != expected_height:

2
hydrus/client/ClientImageHandling.py
View File

@ -63,7 +63,7 @@ def GenerateShapePerceptualHashesNumPy( numpy_image ):
if depth == 4:
# doing this on 10000x10000 pngs eats ram like mad
# we don't want to do GetThumbnailResolutionAndClipRegion as for extremely wide or tall images, we'll then scale below 32 pixels for one dimension, losing information!
# we don't want to do GetThumbnailResolution as for extremely wide or tall images, we'll then scale below 32 pixels for one dimension, losing information!
# however, it does not matter if we stretch the image a bit, since we'll be coercing 32x32 in a minute
new_x = min( 256, x )

28
hydrus/client/ClientPDFHandling.py
View File

@ -75,24 +75,15 @@ def LoadPDF( path: str ):
return document
def GenerateThumbnailNumPyFromPDFPath( path: str, target_resolution: typing.Tuple[int, int], clip_rect = None ) -> bytes:
def GenerateThumbnailNumPyFromPDFPath( path: str, target_resolution: typing.Tuple[int, int] ) -> bytes:
try:
document = LoadPDF( path )
if clip_rect is None:
( target_width, target_height ) = target_resolution
resolution = QC.QSize( target_width, target_height )
else:
( pdf_width, pdf_height ) = GetPDFResolutionFromDocument( document )
resolution = QC.QSize( pdf_width, pdf_height )
( target_width, target_height ) = target_resolution
resolution = QC.QSize( target_width, target_height )
qt_image = document.render(0, resolution)
@ -103,16 +94,7 @@ def GenerateThumbnailNumPyFromPDFPath( path: str, target_resolution: typing.Tupl
document.close()
if clip_rect is None:
thumbnail_numpy_image = numpy_image
else:
numpy_image = HydrusImageHandling.ClipNumPyImage( numpy_image, clip_rect )
thumbnail_numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, target_resolution )
thumbnail_numpy_image = numpy_image
return thumbnail_numpy_image

26
hydrus/client/ClientSVGHandling.py
View File

@ -31,7 +31,7 @@ def LoadSVGRenderer( path: str ):
return renderer
def GenerateThumbnailNumPyFromSVGPath( path: str, target_resolution: typing.Tuple[int, int], clip_rect = None ) -> bytes:
def GenerateThumbnailNumPyFromSVGPath( path: str, target_resolution: typing.Tuple[int, int] ) -> bytes:
# TODO: SVGs have no inherent resolution, so all this is pretty stupid. we should render to exactly the res we want and then clip the result, not beforehand
@ -42,16 +42,9 @@ def GenerateThumbnailNumPyFromSVGPath( path: str, target_resolution: typing.Tupl
# Seems to help for some weird floating point dimension SVGs
renderer.setAspectRatioMode( QC.Qt.AspectRatioMode.KeepAspectRatio )
if clip_rect is None:
( target_width, target_height ) = target_resolution
qt_image = QG.QImage( target_width, target_height, QG.QImage.Format_RGBA8888 )
else:
qt_image = QG.QImage( renderer.defaultSize(), QG.QImage.Format_RGBA8888 )
( target_width, target_height ) = target_resolution
qt_image = QG.QImage( target_width, target_height, QG.QImage.Format_RGBA8888 )
qt_image.fill( QC.Qt.transparent )
@ -63,16 +56,7 @@ def GenerateThumbnailNumPyFromSVGPath( path: str, target_resolution: typing.Tupl
numpy_image = ClientGUIFunctions.ConvertQtImageToNumPy( qt_image )
if clip_rect is None:
thumbnail_numpy_image = numpy_image
else:
numpy_image = HydrusImageHandling.ClipNumPyImage( numpy_image, clip_rect )
thumbnail_numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, target_resolution )
thumbnail_numpy_image = numpy_image
return thumbnail_numpy_image

4
hydrus/client/ClientServices.py
View File

@ -2070,7 +2070,7 @@ class ServiceRepository( ServiceRestricted ):
HG.client_controller.WriteSynchronous( 'schedule_repository_update_file_maintenance', self._service_key, ClientFiles.REGENERATE_FILE_DATA_JOB_FILE_INTEGRITY_PRESENCE_REMOVE_RECORD )
raise Exception( 'An unusual error has occured during repository processing: a definition update file ({}) was missing. Your repository should be paused, and all update files have been scheduled for a presence check. I recommend you run _database->maintenance->clear orphan file records_ too. Please then permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository.'.format( definition_hash.hex() ) )
raise Exception( 'An unusual error has occured during repository processing: a definition update file ({}) was missing. Your repository should be paused, and all update files have been scheduled for a presence check. I recommend you run _database->maintenance->clear/fix orphan file records_ too. Please then permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository.'.format( definition_hash.hex() ) )
with open( update_path, 'rb' ) as f:
@ -2199,7 +2199,7 @@ class ServiceRepository( ServiceRestricted ):
HG.client_controller.WriteSynchronous( 'schedule_repository_update_file_maintenance', self._service_key, ClientFiles.REGENERATE_FILE_DATA_JOB_FILE_INTEGRITY_PRESENCE_REMOVE_RECORD )
raise Exception( 'An unusual error has occured during repository processing: a content update file ({}) was missing. Your repository should be paused, and all update files have been scheduled for a presence check. I recommend you run _database->maintenance->clear orphan file records_ too. Please then permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository.'.format( content_hash.hex() ) )
raise Exception( 'An unusual error has occured during repository processing: a content update file ({}) was missing. Your repository should be paused, and all update files have been scheduled for a presence check. I recommend you run _database->maintenance->clear/fix orphan file records_ too. Please then permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository.'.format( content_hash.hex() ) )
with open( update_path, 'rb' ) as f:

11
hydrus/client/caches/ClientCaches.py
View File

@ -474,7 +474,7 @@ class ThumbnailCache( object ):
thumbnail_scale_type = self._controller.new_options.GetInteger( 'thumbnail_scale_type' )
thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
( clip_rect, ( expected_width, expected_height ) ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( ( media_width, media_height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
( expected_width, expected_height ) = HydrusImageHandling.GetThumbnailResolution( ( media_width, media_height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
numpy_image = HydrusBlurhash.GetNumpyFromBlurhash( blurhash, expected_width, expected_height )
@ -565,7 +565,7 @@ class ThumbnailCache( object ):
thumbnail_scale_type = self._controller.new_options.GetInteger( 'thumbnail_scale_type' )
thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
( clip_rect, ( expected_width, expected_height ) ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( ( media_width, media_height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
( expected_width, expected_height ) = HydrusImageHandling.GetThumbnailResolution( ( media_width, media_height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
exactly_as_expected = current_width == expected_width and current_height == expected_height
@ -806,12 +806,7 @@ class ThumbnailCache( object ):
numpy_image_resolution = HydrusImageHandling.GetResolutionNumPy( numpy_image )
( clip_rect, target_resolution ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( numpy_image_resolution, bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
if clip_rect is not None:
numpy_image = HydrusImageHandling.ClipNumPyImage( numpy_image, clip_rect )
target_resolution = HydrusImageHandling.GetThumbnailResolution( numpy_image_resolution, bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, target_resolution )

472
hydrus/client/db/ClientDB.py
View File

@ -1056,7 +1056,7 @@ class DB( HydrusDB.HydrusDB ):
job_key = ClientThreading.JobKey( cancellable = True )
job_key.SetStatusTitle( 'clear orphan file records' )
job_key.SetStatusTitle( 'clear/fix orphan file records' )
self._controller.pub( 'modal_message', job_key )
@ -1066,9 +1066,10 @@ class DB( HydrusDB.HydrusDB ):
job_key.SetStatusText( 'looking for orphans' )
# actually important we do it in this order I guess, to potentially fix a file that is only in 'my files' and not in 'all my files' or 'all local files'
jobs = [
( ( HC.LOCAL_FILE_DOMAIN, ), self.modules_services.combined_local_media_service_id, 'my files umbrella' ),
( ( HC.LOCAL_FILE_TRASH_DOMAIN, HC.COMBINED_LOCAL_MEDIA, HC.LOCAL_FILE_UPDATE_DOMAIN, ), self.modules_services.combined_local_file_service_id, 'local files umbrella' )
( ( HC.LOCAL_FILE_DOMAIN, ), self.modules_services.combined_local_media_service_id, 'all my files umbrella' ),
( ( HC.LOCAL_FILE_TRASH_DOMAIN, HC.COMBINED_LOCAL_MEDIA, HC.LOCAL_FILE_UPDATE_DOMAIN, ), self.modules_services.combined_local_file_service_id, 'all local files umbrella' )
]
for ( umbrella_components_service_types, umbrella_master_service_id, description ) in jobs:
@ -1092,31 +1093,118 @@ class DB( HydrusDB.HydrusDB ):
return
job_key.SetStatusText( 'deleting orphans' )
job_key.SetStatusText( 'actioning orphans' )
if len( in_components_not_in_master ) > 0:
orphans_found = True
# these files were deleted from the umbrella service without being cleared from a specific file domain
# they are most likely deleted from disk
# pushing the master's delete call will flush from the components as well
client_files_manager = HG.client_controller.client_files_manager
self._DeleteFiles( umbrella_master_service_id, in_components_not_in_master )
those_that_exist_on_disk = set()
# we spam this stuff since it won't trigger if the files don't exist on master!
self.modules_files_inbox.ArchiveFiles( in_components_not_in_master )
hash_ids_to_hashes = self.modules_hashes_local_cache.GetHashIdsToHashes( hash_ids = in_components_not_in_master )
for hash_id in in_components_not_in_master:
for ( hash_id, hash ) in hash_ids_to_hashes.items():
self.modules_similar_files.StopSearchingFile( hash_id )
try:
mime = self.modules_files_metadata_basic.GetMime( hash_id )
except HydrusExceptions.DataMissing:
continue
if client_files_manager.LocklessHasFile( hash, mime ):
those_that_exist_on_disk.add( hash_id )
self.modules_files_maintenance_queue.CancelFiles( in_components_not_in_master )
those_that_are_missing = set( in_components_not_in_master ).difference( those_that_exist_on_disk )
self.modules_hashes_local_cache.DropHashIdsFromCache( in_components_not_in_master )
if len( those_that_exist_on_disk ) > 0:
# ok these we actually have but they aren't listed on the umbrella service. sounds like an import that went wrong
# it would be nice to recover these files to save the import timestamp, but in the same stroke they may be borked deletes so we want to present them to the user
import_rows = []
for hash_id in those_that_exist_on_disk:
timestamps = []
for umbrella_components_service_id in umbrella_components_service_ids:
service_key = self.modules_services.GetServiceKey( umbrella_components_service_id )
timestamp_data = ClientTime.TimestampData( HC.TIMESTAMP_TYPE_IMPORTED, location = service_key )
timestamp = self.modules_files_storage.GetTimestamp( hash_id, timestamp_data )
if timestamp is not None:
timestamps.append( timestamp )
if len( timestamps ) == 0:
those_that_are_missing.add( hash_id )
else:
timestamp = min( timestamps )
import_rows.append( ( hash_id, timestamp ) )
if len( import_rows ) > 0:
# with fingers crossed this magically corrects all sorts of stuff
self._AddFiles( umbrella_master_service_id, import_rows )
HydrusData.ShowText( 'Found and recovered {} records for files that were safely in specific component services components but not the master "{}". I have opened a new page with these files--they may have been faulty imports or faulty deletes, so you probably need to give them a look.'.format( HydrusData.ToHumanInt( len( in_components_not_in_master ) ), description ) )
service_key = self.modules_services.GetServiceKey( umbrella_master_service_id )
location_context = ClientLocation.LocationContext.STATICCreateSimple( service_key )
hashes = self.modules_hashes_local_cache.GetHashes( [ row[0] for row in import_rows ] )
HG.client_controller.pub( 'new_page_query', location_context, initial_hashes = hashes, page_name = 'reparented file records' )
HydrusData.ShowText( 'Found and deleted {} files that were in components but not the master {}.'.format( HydrusData.ToHumanInt( len( in_components_not_in_master ) ), description ) )
if len( those_that_are_missing ) > 0:
# these files were deleted from the umbrella service without being cleared from a specific file domain
# they are most likely deleted from disk
# we'll spam our delete calls
self._DeleteFiles( umbrella_master_service_id, in_components_not_in_master )
for umbrella_components_service_id in umbrella_components_service_ids:
self._DeleteFiles( umbrella_components_service_id, in_components_not_in_master )
# we spam this stuff since it won't trigger if the files don't exist on master!
self.modules_files_inbox.ArchiveFiles( in_components_not_in_master )
for hash_id in in_components_not_in_master:
self.modules_similar_files.StopSearchingFile( hash_id )
self.modules_files_maintenance_queue.CancelFiles( in_components_not_in_master )
self.modules_hashes_local_cache.DropHashIdsFromCache( in_components_not_in_master )
HydrusData.ShowText( 'Found and deleted {} records for files that were in specific service components but not the master "{}".'.format( HydrusData.ToHumanInt( len( in_components_not_in_master ) ), description ) )
if job_key.IsCancelled():
@ -1134,7 +1222,7 @@ class DB( HydrusDB.HydrusDB ):
self._DeleteFiles( umbrella_master_service_id, in_master_not_in_components )
HydrusData.ShowText( 'Found and deleted {} files that were in the master {} but not it its components.'.format( HydrusData.ToHumanInt( len( in_master_not_in_components ) ), description ) )
HydrusData.ShowText( 'Found and deleted {} records for files that were in the master "{}" but not it its specific service components.'.format( HydrusData.ToHumanInt( len( in_master_not_in_components ) ), description ) )
@ -8361,358 +8449,6 @@ class DB( HydrusDB.HydrusDB ):
self._controller.frame_splash_status.SetText( 'updating db to v' + str( version + 1 ) )
if version == 480:
try:
from hydrus.client.gui.canvas import ClientGUIMPV
if ClientGUIMPV.MPV_IS_AVAILABLE and HC.PLATFORM_LINUX:
new_options = self.modules_serialisable.GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_CLIENT_OPTIONS )
show_message = False
for mime in ( HC.ANIMATION_GIF, HC.VIDEO_MP4, HC.AUDIO_MP3 ):
( media_show_action, media_start_paused, media_start_with_embed ) = new_options.GetMediaShowAction( mime )
if media_show_action == CC.MEDIA_VIEWER_ACTION_SHOW_WITH_NATIVE:
show_message = True
if show_message:
message = 'Hey, you are a Linux user and seem to have MPV support but you are not set to use MPV for one or more filetypes. If you know all about this, no worries, ignore this message. But if you are a long-time Linux user, you may have been reverted to the native hydrus renderer many releases ago due to stability worries. If you did not know hydrus supports audio now, please check the filetype options under _options->media_ and give mpv a go!'
self.pub_initial_message( message )
except:
pass
if version == 481:
try:
new_options = self.modules_serialisable.GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_CLIENT_OPTIONS )
old_options = self._GetOptions()
new_options.SetInteger( 'thumbnail_cache_size', old_options[ 'thumbnail_cache_size' ] )
new_options.SetInteger( 'image_cache_size', old_options[ 'fullscreen_cache_size' ] )
new_options.SetBoolean( 'pause_export_folders_sync', old_options[ 'pause_export_folders_sync' ] )
new_options.SetBoolean( 'pause_import_folders_sync', old_options[ 'pause_import_folders_sync' ] )
new_options.SetBoolean( 'pause_repo_sync', old_options[ 'pause_repo_sync' ] )
new_options.SetBoolean( 'pause_subs_sync', old_options[ 'pause_subs_sync' ] )
self.modules_serialisable.SetJSONDump( new_options )
except Exception as e:
HydrusData.PrintException( e )
message = 'Updating some cache sizes and pause states to a new options structure failed! This is not super important, but hydev would be interested in seeing the error that was printed to the log. Also check _options->speed and memory_ for your thumbnail/image cache sizes, and your subs/repository/import folder/export folder pause status.'
self.pub_initial_message( message )
if version == 482:
self._Execute( 'UPDATE services SET service_type = ? WHERE service_key = ?;', ( HC.LOCAL_FILE_UPDATE_DOMAIN, sqlite3.Binary( CC.LOCAL_UPDATE_SERVICE_KEY ) ) )
try:
domain_manager = self.modules_serialisable.GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER )
domain_manager.Initialise()
#
domain_manager.OverwriteDefaultParsers( [
'nijie view popup parser',
'deviant art file extended_fetch parser'
] )
#
domain_manager.OverwriteDefaultURLClasses( [
'mega.nz file or folder',
'mega.nz file',
'mega.nz folder (alt format)',
'mega.nz folder'
] )
#
domain_manager.TryToLinkURLClassesAndParsers()
#
self.modules_serialisable.SetJSONDump( domain_manager )
except Exception as e:
HydrusData.PrintException( e )
message = 'Trying to update some parsers failed! Please let hydrus dev know!'
self.pub_initial_message( message )
if version == 485:
result = self._Execute( 'SELECT service_id FROM services WHERE service_type = ?;', ( HC.COMBINED_LOCAL_MEDIA, ) ).fetchone()
if result is None:
warning_ptr_text = 'After looking at your database, I think this will be quick, maybe a couple minutes at most.'
nums_mappings = self._STL( self._Execute( 'SELECT info FROM service_info WHERE info_type = ?;', ( HC.SERVICE_INFO_NUM_MAPPINGS, ) ) )
if len( nums_mappings ) > 0:
we_ptr = max( nums_mappings ) > 1000000000
if we_ptr:
result = self._Execute( 'SELECT info FROM service_info WHERE info_type = ? AND service_id = ?;', ( HC.SERVICE_INFO_NUM_FILES, self.modules_services.combined_local_file_service_id ) ).fetchone()
if result is not None:
( num_files, ) = result
warning_ptr_text = 'For most users, this update works at about 25-100k files per minute, so with {} files I expect it to take ~{} minutes for you.'.format( HydrusData.ToHumanInt( num_files ), max( 1, int( num_files / 60000 ) ) )
else:
we_ptr = False
else:
we_ptr = False
message = 'Your database is going to calculate some new data so it can refer to multiple local services more efficiently. It could take a while.'
message += os.linesep * 2
message += warning_ptr_text
message += os.linesep * 2
message += 'If you do not have the time at the moment, please force kill the hydrus process now. Otherwise, continue!'
BlockingSafeShowMessage( message )
client_caches_path = os.path.join( self._db_dir, 'client.caches.db' )
expected_space_needed = os.path.getsize( client_caches_path ) // 4
try:
HydrusDBBase.CheckHasSpaceForDBTransaction( self._db_dir, expected_space_needed )
except Exception as e:
message = 'Hey, this update is going to expand your database cache. It requires some free space, but I think there is a problem and I am not sure it can be done safely. I recommend you kill the hydrus process now and free up some space. If you think the check is mistaken, click ok and it will try anyway. Full error:'
message += os.linesep * 2
message += str( e )
BlockingSafeShowMessage( message )
self._controller.frame_splash_status.SetText( 'creating "all my files" virtual service' )
self._controller.frame_splash_status.SetSubtext( 'gathering current file records' )
self._cursor_transaction_wrapper.Commit()
self._Execute( 'PRAGMA journal_mode = TRUNCATE;' )
self._cursor_transaction_wrapper.BeginImmediate()
dictionary = ClientServices.GenerateDefaultServiceDictionary( HC.COMBINED_LOCAL_MEDIA )
self._AddService( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY, HC.COMBINED_LOCAL_MEDIA, 'all my files', dictionary )
self._UnloadModules()
self._LoadModules()
# services module is now aware of the new guy
# note we do not have to populate the mappings cache--we just have to add files naturally!
# current files
all_media_hash_ids = set()
for service_id in self.modules_services.GetServiceIds( ( HC.LOCAL_FILE_DOMAIN, ) ):
all_media_hash_ids.update( self.modules_files_storage.GetCurrentHashIdsList( service_id ) )
num_to_do = len( all_media_hash_ids )
BLOCK_SIZE = 500
for ( i, block_of_hash_ids ) in enumerate( HydrusData.SplitIteratorIntoChunks( all_media_hash_ids, BLOCK_SIZE ) ):
block_of_hash_ids_to_timestamps = self.modules_files_storage.GetCurrentHashIdsToTimestamps( self.modules_services.combined_local_file_service_id, block_of_hash_ids )
rows = list( block_of_hash_ids_to_timestamps.items() )
self._AddFiles( self.modules_services.combined_local_media_service_id, rows )
self._controller.frame_splash_status.SetSubtext( 'making current file records: {}'.format( HydrusData.ConvertValueRangeToPrettyString( i * BLOCK_SIZE, num_to_do ) ) )
# deleted files
self._controller.frame_splash_status.SetSubtext( 'gathering deleted file records' )
all_media_hash_ids = set()
hash_ids_to_deletion_timestamps = {}
for service_id in self.modules_services.GetServiceIds( ( HC.LOCAL_FILE_DOMAIN, ) ):
deleted_files_table_name = ClientDBFilesStorage.GenerateFilesTableName( self.modules_services.combined_local_file_service_id, HC.CONTENT_STATUS_DELETED )
results = self._Execute( 'SELECT hash_id, timestamp FROM {};'.format( deleted_files_table_name ) ).fetchall()
for ( hash_id, timestamp ) in results:
all_media_hash_ids.add( hash_id )
if timestamp is not None:
if hash_id in hash_ids_to_deletion_timestamps:
hash_ids_to_deletion_timestamps[ hash_id ] = max( timestamp, hash_ids_to_deletion_timestamps[ hash_id ] )
else:
hash_ids_to_deletion_timestamps[ hash_id ] = timestamp
deleted_files_table_name = ClientDBFilesStorage.GenerateFilesTableName( self.modules_services.combined_local_file_service_id, HC.CONTENT_STATUS_DELETED )
hash_ids_to_original_timestamps = dict( self._Execute( 'SELECT hash_id, original_timestamp FROM {};'.format( deleted_files_table_name ) ) )
current_files_table_name = ClientDBFilesStorage.GenerateFilesTableName( self.modules_services.combined_local_file_service_id, HC.CONTENT_STATUS_CURRENT )
hash_ids_to_original_timestamps.update( dict( self._Execute( 'SELECT hash_id, timestamp FROM {};'.format( current_files_table_name ) ) ) )
deleted_files_table_name = ClientDBFilesStorage.GenerateFilesTableName( self.modules_services.combined_local_media_service_id, HC.CONTENT_STATUS_DELETED )
num_to_do = len( all_media_hash_ids )
for ( i, hash_id ) in enumerate( all_media_hash_ids ):
# no need to fake the service info number updates--that will calculate from raw on next review services open
if hash_id not in hash_ids_to_deletion_timestamps:
timestamp = None
else:
timestamp = hash_ids_to_deletion_timestamps[ hash_id ]
if hash_id not in hash_ids_to_original_timestamps:
continue
else:
original_timestamp = hash_ids_to_original_timestamps[ hash_id ]
self._Execute( 'INSERT OR IGNORE INTO {} ( hash_id, timestamp, original_timestamp ) VALUES ( ?, ?, ? );'.format( deleted_files_table_name ), ( hash_id, timestamp, original_timestamp ) )
if i % 500 == 0:
self._controller.frame_splash_status.SetSubtext( 'making deleted file records: {}'.format( HydrusData.ConvertValueRangeToPrettyString( i, num_to_do ) ) )
self._cursor_transaction_wrapper.Commit()
self._Execute( 'PRAGMA journal_mode = {};'.format( HG.db_journal_mode ) )
self._cursor_transaction_wrapper.BeginImmediate()
self._controller.frame_splash_status.SetSubtext( '' )
if version == 486:
file_service_ids = self.modules_services.GetServiceIds( HC.FILE_SERVICES_WITH_SPECIFIC_MAPPING_CACHES )
tag_service_ids = self.modules_services.GetServiceIds( HC.REAL_TAG_SERVICES )
for ( file_service_id, tag_service_id ) in itertools.product( file_service_ids, tag_service_ids ):
# some users still have a few of these floating around, they are not needed
suffix = '{}_{}'.format( file_service_id, tag_service_id )
cache_files_table_name = 'external_caches.specific_files_cache_{}'.format( suffix )
result = self._Execute( 'SELECT 1 FROM external_caches.sqlite_master WHERE name = ?;', ( cache_files_table_name.split( '.', 1 )[1], ) ).fetchone()
if result is None:
continue
self._Execute( 'DROP TABLE {};'.format( cache_files_table_name ) )
if version == 488:
# clearing up some garbo 1970-01-01 timestamps that got saved
self._Execute( 'DELETE FROM file_domain_modified_timestamps WHERE file_modified_timestamp < ?;', ( 86400 * 7, ) )
#
# mysterious situation where repo updates domain had some ghost files that were not in all local files!
hash_ids_in_repo_updates = set( self.modules_files_storage.GetCurrentHashIdsList( self.modules_services.local_update_service_id ) )
hash_ids_in_all_files = self.modules_files_storage.FilterHashIds( ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_FILE_SERVICE_KEY ), hash_ids_in_repo_updates )
orphan_hash_ids = hash_ids_in_repo_updates.difference( hash_ids_in_all_files )
if len( orphan_hash_ids ) > 0:
hash_ids_to_timestamps = self.modules_files_storage.GetCurrentHashIdsToTimestamps( self.modules_services.local_update_service_id, orphan_hash_ids )
rows = list( hash_ids_to_timestamps.items() )
self.modules_files_storage.AddFiles( self.modules_services.combined_local_file_service_id, rows )
# turns out ffmpeg was detecting some updates as mpegs, so this wasn't always working right!
self.modules_files_maintenance_queue.AddJobs( hash_ids_in_repo_updates, ClientFiles.REGENERATE_FILE_DATA_JOB_FILE_METADATA )
self._Execute( 'DELETE FROM service_info WHERE service_id = ?;', ( self.modules_services.local_update_service_id, ) )
if version == 490:
try:

6
hydrus/client/gui/ClientGUI.py
View File

@ -1212,11 +1212,11 @@ class FrameGUI( CAC.ApplicationCommandProcessorMixin, ClientGUITopLevelWindows.M
text = 'DO NOT RUN THIS UNLESS YOU KNOW YOU NEED TO'
text += os.linesep * 2
text += 'This will instruct the database to review its file records\' integrity. If anything appears to be in a specific domain (e.g. my files) but not an umbrella domain (e.g. all my files), or vice versa, it will be removed.'
text += 'This will instruct the database to review its file records\' integrity. If anything appears to be in a specific domain (e.g. my files) but not an umbrella domain (e.g. all my files), and the actual file also exists on disk, it will try to recover the record. If the file does not actually exist on disk, or the record is in the umbrella domain and not in the specific domain, or if recovery data cannot be found, the record will be deleted.'
text += os.linesep * 2
text += 'You typically do not ever see these files and they are basically harmless, but they can offset some file counts confusingly and may break other maintenance routines. You probably only need to run this if you can\'t process the apparent last handful of duplicate filter pairs or hydrus dev otherwise told you to try it.'
text += os.linesep * 2
text += 'It will create a popup message while it works and inform you of the number of orphan records found.'
text += 'It will create a popup message while it works and inform you of the number of orphan records found. It may lock up the client for a bit.'
result = ClientGUIDialogsQuick.GetYesNo( self, text, yes_label = 'do it', no_label = 'forget it' )
@ -3111,7 +3111,7 @@ class FrameGUI( CAC.ApplicationCommandProcessorMixin, ClientGUITopLevelWindows.M
ClientGUIMenus.AppendSeparator( db_maintenance_submenu )
ClientGUIMenus.AppendMenuItem( db_maintenance_submenu, 'clear orphan file records', 'Clear out surplus file records that have not been deleted correctly.', self._ClearOrphanFileRecords )
ClientGUIMenus.AppendMenuItem( db_maintenance_submenu, 'clear/fix orphan file records', 'Clear out surplus file records that have not been deleted correctly.', self._ClearOrphanFileRecords )
ClientGUIMenus.AppendMenuItem( db_maintenance_submenu, 'clear orphan tables', 'Clear out surplus db tables that have not been deleted correctly.', self._ClearOrphanTables )

8
hydrus/client/gui/ClientGUIDialogsQuick.py
View File

@ -23,16 +23,16 @@ def GetDeleteFilesJobs( win, media, default_reason, suggested_file_service_key =
if panel.QuestionIsAlreadyResolved():
( involves_physical_delete, jobs ) = panel.GetValue()
( hashes_physically_deleted, jobs ) = panel.GetValue()
return ( involves_physical_delete, jobs )
return ( hashes_physically_deleted, jobs )
if dlg.exec() == QW.QDialog.Accepted:
( involves_physical_delete, jobs ) = panel.GetValue()
( hashes_physically_deleted, jobs ) = panel.GetValue()
return ( involves_physical_delete, jobs )
return ( hashes_physically_deleted, jobs )
else:

26
hydrus/client/gui/ClientGUIScrolledPanelsEdit.py
View File

@ -685,7 +685,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
#
( file_service_key, list_of_service_keys_to_content_updates, save_reason, involves_physical_delete, description ) = self._action_radio.GetValue()
( file_service_key, list_of_service_keys_to_content_updates, save_reason, hashes_physically_deleted, description ) = self._action_radio.GetValue()
self._simple_description.setText( description )
@ -888,14 +888,14 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
save_reason = True
involves_physical_delete = False
hashes_physically_deleted = []
num_local_services_done += 1
# this is an ugly place to put this, and the mickey-mouse append, but it works
if self._num_actionable_local_file_services > 1 and num_local_services_done == self._num_actionable_local_file_services:
self._permitted_action_choices.append( ( text, ( deletee_file_service_key, list_of_service_keys_to_content_updates, save_reason, involves_physical_delete, text ) ) )
self._permitted_action_choices.append( ( text, ( deletee_file_service_key, list_of_service_keys_to_content_updates, save_reason, hashes_physically_deleted, text ) ) )
deletee_file_service_key = CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY
@ -911,7 +911,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
save_reason = True
involves_physical_delete = False
hashes_physically_deleted = []
elif deletee_service_type == HC.FILE_REPOSITORY:
@ -948,7 +948,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
list_of_service_keys_to_content_updates = [ { deletee_file_service_key : content_updates } ]
involves_physical_delete = False
hashes_physically_deleted = []
elif deletee_file_service_key == CC.COMBINED_LOCAL_FILE_SERVICE_KEY:
@ -985,10 +985,10 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
save_reason = True
involves_physical_delete = True
hashes_physically_deleted = hashes
self._permitted_action_choices.append( ( text, ( deletee_file_service_key, list_of_service_keys_to_content_updates, save_reason, involves_physical_delete, text ) ) )
self._permitted_action_choices.append( ( text, ( deletee_file_service_key, list_of_service_keys_to_content_updates, save_reason, hashes_physically_deleted, text ) ) )
if self._num_actionable_local_file_services == 1 and not HC.options[ 'confirm_trash' ]:
@ -1028,9 +1028,9 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
save_reason = False
involves_physical_delete = True
hashes_physically_deleted = hashes
self._permitted_action_choices.append( ( text, ( 'clear_delete', list_of_service_keys_to_content_updates, save_reason, involves_physical_delete, text ) ) )
self._permitted_action_choices.append( ( text, ( 'clear_delete', list_of_service_keys_to_content_updates, save_reason, hashes_physically_deleted, text ) ) )
@ -1092,7 +1092,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
def _UpdateControls( self ):
( file_service_key, list_of_service_keys_to_content_updates, save_reason, involves_physical_delete, description ) = self._action_radio.GetValue()
( file_service_key, list_of_service_keys_to_content_updates, save_reason, hashes_physically_deleted, description ) = self._action_radio.GetValue()
reason_permitted = save_reason
@ -1124,9 +1124,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
return ( False, [] )
involves_physical_delete = False
( file_service_key, list_of_service_keys_to_content_updates, save_reason, involves_physical_delete, description ) = self._action_radio.GetValue()
( file_service_key, list_of_service_keys_to_content_updates, save_reason, hashes_physically_deleted, description ) = self._action_radio.GetValue()
if save_reason:
@ -1190,7 +1188,7 @@ class EditDeleteFilesPanel( ClientGUIScrolledPanels.EditPanel ):
return ( involves_physical_delete, list_of_service_keys_to_content_updates )
return ( hashes_physically_deleted, list_of_service_keys_to_content_updates )
def QuestionIsAlreadyResolved( self ):

81
hydrus/client/gui/ClientGUIScrolledPanelsManagement.py
View File

@ -5027,34 +5027,72 @@ class ManageURLsPanel( CAC.ApplicationCommandProcessorMixin, ClientGUIScrolledPa
HG.client_controller.pub( 'clipboard', 'text', text )
def _EnterURL( self, url, only_add = False ):
def _EnterURLs( self, urls, only_add = False ):
normalised_url = HG.client_controller.network_engine.domain_manager.NormaliseURL( url )
normalised_urls = []
weird_urls = []
addee_media = set()
for m in self._current_media:
for url in urls:
locations_manager = m.GetLocationsManager()
if normalised_url not in locations_manager.GetURLs():
try:
addee_media.add( m )
normalised_url = HG.client_controller.network_engine.domain_manager.NormaliseURL( url )
normalised_urls.append( normalised_url )
except HydrusExceptions.URLClassException:
weird_urls.append( url )
if len( addee_media ) > 0:
if len( weird_urls ) > 0:
addee_hashes = { m.GetHash() for m in addee_media }
message = 'The URLs:'
message += '\n' * 2
message += '\n'.join( weird_urls )
message += '\n' * 2
message += '--did not parse. Normally I would not recommend importing invalid URLs, but do you want to force it anyway?'
content_update = HydrusData.ContentUpdate( HC.CONTENT_TYPE_URLS, HC.CONTENT_UPDATE_ADD, ( ( normalised_url, ), addee_hashes ) )
result = ClientGUIDialogsQuick.GetYesNo( self, message )
for m in addee_media:
if result != QW.QDialog.Accepted:
m.GetMediaResult().ProcessContentUpdate( CC.COMBINED_LOCAL_FILE_SERVICE_KEY, content_update )
return False
self._pending_content_updates.append( content_update )
normalised_urls.extend( weird_urls )
normalised_urls = HydrusData.DedupeList( normalised_urls )
for normalised_url in normalised_urls:
addee_media = set()
for m in self._current_media:
locations_manager = m.GetLocationsManager()
if normalised_url not in locations_manager.GetURLs():
addee_media.add( m )
if len( addee_media ) > 0:
addee_hashes = { m.GetHash() for m in addee_media }
content_update = HydrusData.ContentUpdate( HC.CONTENT_TYPE_URLS, HC.CONTENT_UPDATE_ADD, ( ( normalised_url, ), addee_hashes ) )
for m in addee_media:
m.GetMediaResult().ProcessContentUpdate( CC.COMBINED_LOCAL_FILE_SERVICE_KEY, content_update )
self._pending_content_updates.append( content_update )
#
@ -5077,13 +5115,9 @@ class ManageURLsPanel( CAC.ApplicationCommandProcessorMixin, ClientGUIScrolledPa
try:
for url in HydrusText.DeserialiseNewlinedTexts( raw_text ):
if url != '':
self._EnterURL( url, only_add = True )
urls = HydrusText.DeserialiseNewlinedTexts( raw_text )
self._EnterURLs( urls, only_add = True )
except Exception as e:
@ -5195,6 +5229,7 @@ class ManageURLsPanel( CAC.ApplicationCommandProcessorMixin, ClientGUIScrolledPa
else:
return True # was: event.ignore()
def AddURL( self ):
@ -5209,7 +5244,7 @@ class ManageURLsPanel( CAC.ApplicationCommandProcessorMixin, ClientGUIScrolledPa
try:
self._EnterURL( url )
self._EnterURLs( [ url ] )
self._url_input.clear()

14
hydrus/client/gui/canvas/ClientGUICanvas.py
View File

@ -461,7 +461,7 @@ class Canvas( CAC.ApplicationCommandProcessorMixin, QW.QWidget ):
try:
( involves_physical_delete, jobs ) = ClientGUIDialogsQuick.GetDeleteFilesJobs( self, media, default_reason, suggested_file_service_key = file_service_key )
( hashes_physically_deleted, jobs ) = ClientGUIDialogsQuick.GetDeleteFilesJobs( self, media, default_reason, suggested_file_service_key = file_service_key )
except HydrusExceptions.CancelledException:
@ -912,9 +912,9 @@ class Canvas( CAC.ApplicationCommandProcessorMixin, QW.QWidget ):
if action == CAC.SIMPLE_COPY_LITTLE_BMP and ( width > 1024 or height > 1024 ):
( clip_rect, clipped_res ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( self._current_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
target_resolution = HydrusImageHandling.GetThumbnailResolution( self._current_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
copied = self._CopyBMPToClipboard( resolution = clipped_res )
copied = self._CopyBMPToClipboard( resolution = target_resolution )
else:
@ -1587,9 +1587,9 @@ class CanvasPanel( Canvas ):
if width > 1024 or height > 1024:
( clip_rect, clipped_res ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( self._current_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
target_resolution = HydrusImageHandling.GetThumbnailResolution( self._current_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
ClientGUIMenus.AppendMenuItem( copy_menu, 'source lookup bitmap ({}x{})'.format( clipped_res[0], clipped_res[1] ), 'Copy a smaller bitmap of this file, for quicker lookup on source-finding websites.', self._CopyBMPToClipboard, clipped_res )
ClientGUIMenus.AppendMenuItem( copy_menu, 'source lookup bitmap ({}x{})'.format( target_resolution[0], target_resolution[1] ), 'Copy a smaller bitmap of this file, for quicker lookup on source-finding websites.', self._CopyBMPToClipboard, target_resolution )
@ -4604,9 +4604,9 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ):
if width > 1024 or height > 1024:
( clip_rect, clipped_res ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( self._current_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
target_resolution = HydrusImageHandling.GetThumbnailResolution( self._current_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
ClientGUIMenus.AppendMenuItem( copy_menu, 'source lookup bitmap ({}x{})'.format( clipped_res[0], clipped_res[1] ), 'Copy a smaller bitmap of this file, for quicker lookup on source-finding websites.', self._CopyBMPToClipboard, clipped_res )
ClientGUIMenus.AppendMenuItem( copy_menu, 'source lookup bitmap ({}x{})'.format( target_resolution[0], target_resolution[1] ), 'Copy a smaller bitmap of this file, for quicker lookup on source-finding websites.', self._CopyBMPToClipboard, target_resolution )

10
hydrus/client/gui/canvas/ClientGUICanvasMedia.py
View File

@ -55,6 +55,7 @@ zoom_centerpoints_str_lookup[ ZOOM_CENTERPOINT_MOUSE ] = 'mouse (or viewer cente
zoom_centerpoints_str_lookup[ ZOOM_CENTERPOINT_MEDIA_TOP_LEFT ] = 'media top-left'
OPEN_EXTERNALLY_BUTTON_SIZE = ( 200, 45 )
OPEN_EXTERNALLY_MAX_THUMBNAIL_SIZE = ( 200, 200 )
def CalculateCanvasMediaSize( media, canvas_size: QC.QSize, show_action ):
@ -216,9 +217,9 @@ def CalculateMediaContainerSize( media, device_pixel_ratio: float, zoom, show_ac
#thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
thumbnail_dpr_percent = 100
( clip_rect, ( thumb_width, thumb_height ) ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( media.GetResolution(), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
( thumb_width, thumb_height ) = HydrusImageHandling.GetThumbnailResolution( media.GetResolution(), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
height = height + thumb_height
height = height + min( OPEN_EXTERNALLY_MAX_THUMBNAIL_SIZE[1], thumb_height )
return QC.QSize( width, height )
@ -2899,6 +2900,11 @@ class OpenExternallyPanel( QW.QWidget ):
thumbnail_window = ClientGUICommon.BufferedWindowIcon( self, qt_pixmap )
thumbnail_window_width = min( thumbnail_window.width(), OPEN_EXTERNALLY_MAX_THUMBNAIL_SIZE[0] )
thumbnail_window_height = min( thumbnail_window.height(), OPEN_EXTERNALLY_MAX_THUMBNAIL_SIZE[1] )
thumbnail_window.setFixedSize( thumbnail_window_width, thumbnail_window_height )
QP.AddToLayout( vbox, thumbnail_window, CC.FLAGS_CENTER )

18
hydrus/client/gui/lists/ClientGUIListBoxes.py
View File

@ -1860,9 +1860,16 @@ class ListBox( QW.QScrollArea ):
return
for term in removable_terms:
if len( removable_terms ) > len( self._ordered_terms ) ** 0.5:
self._ordered_terms.remove( term )
self._ordered_terms = [ term for term in self._ordered_terms if term not in removable_terms ]
else:
for term in removable_terms:
self._ordered_terms.remove( term )
self._selected_terms.difference_update( removable_terms )
@ -3889,8 +3896,6 @@ class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
def _UpdateTerms( self, limit_to_these_tags = None ):
# TODO: optimisation here--instead of remove/add, edit the terms _in place_ with new counts, removing now empty, and then just resort
previous_selected_terms = set( self._selected_terms )
if limit_to_these_tags is None:
@ -3935,10 +3940,11 @@ class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
nonzero_terms = [ self._GenerateTermFromTag( tag ) for tag in nonzero_tags ]
if len( removee_terms ) > len( self._ordered_terms ) / 2:
if len( removee_terms ) > len( self._selected_terms ) ** 0.5:
self._Clear()
removee_terms = []
altered_terms = []
new_terms = nonzero_terms
@ -3970,8 +3976,6 @@ class ListBoxTagsMedia( ListBoxTagsDisplayCapable ):
if len( new_terms ) > 0:
# TODO: if not sort_needed at this stage, it would be ideal to call an auto-sorting '_InsertTerms', if that is reasonably doable
self._AppendTerms( new_terms )
sort_needed = True

18
hydrus/client/gui/pages/ClientGUIResults.py
View File

@ -365,16 +365,16 @@ class MediaPanel( CAC.ApplicationCommandProcessorMixin, ClientMedia.ListeningMed
try:
( involves_physical_delete, jobs ) = ClientGUIDialogsQuick.GetDeleteFilesJobs( self, media_to_delete, default_reason, suggested_file_service_key = file_service_key )
( hashes_physically_deleted, jobs ) = ClientGUIDialogsQuick.GetDeleteFilesJobs( self, media_to_delete, default_reason, suggested_file_service_key = file_service_key )
except HydrusExceptions.CancelledException:
return
if involves_physical_delete:
if len( hashes_physically_deleted ) > 0:
self._SetFocusedMedia( None )
self._RemoveMediaByHashes( hashes_physically_deleted )
def do_it( jobs ):
@ -2111,9 +2111,9 @@ class MediaPanel( CAC.ApplicationCommandProcessorMixin, ClientMedia.ListeningMed
if action == CAC.SIMPLE_COPY_LITTLE_BMP and ( width > 1024 or height > 1024 ):
( clip_rect, clipped_res ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( self._focused_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
target_resolution = HydrusImageHandling.GetThumbnailResolution( self._focused_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
copied = self._CopyBMPToClipboard( resolution = clipped_res )
copied = self._CopyBMPToClipboard( resolution = target_resolution )
else:
@ -4049,7 +4049,9 @@ class MediaPanelThumbnails( MediaPanel ):
ClientGUIMenus.AppendSeparator( menu )
local_file_service_keys_we_are_in = sorted( current_file_service_keys.intersection( local_media_file_service_keys ), key = HG.client_controller.services_manager.GetName )
user_command_deletable_file_service_keys = local_media_file_service_keys.union( [ CC.LOCAL_UPDATE_SERVICE_KEY ] )
local_file_service_keys_we_are_in = sorted( current_file_service_keys.intersection( user_command_deletable_file_service_keys ), key = HG.client_controller.services_manager.GetName )
for file_service_key in local_file_service_keys_we_are_in:
@ -4375,9 +4377,9 @@ class MediaPanelThumbnails( MediaPanel ):
if width > 1024 or height > 1024:
( clip_rect, clipped_res ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( self._focused_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
target_resolution = HydrusImageHandling.GetThumbnailResolution( self._focused_media.GetResolution(), ( 1024, 1024 ), HydrusImageHandling.THUMBNAIL_SCALE_TO_FIT, 100 )
ClientGUIMenus.AppendMenuItem( copy_menu, 'source lookup bitmap ({}x{})'.format( clipped_res[0], clipped_res[1] ), 'Copy a smaller bitmap of this file, for quicker lookup on source-finding websites.', self._CopyBMPToClipboard, clipped_res )
ClientGUIMenus.AppendMenuItem( copy_menu, 'source lookup bitmap ({}x{})'.format( target_resolution[0], target_resolution[1] ), 'Copy a smaller bitmap of this file, for quicker lookup on source-finding websites.', self._CopyBMPToClipboard, target_resolution )

7
hydrus/client/gui/widgets/ClientGUICommon.py
View File

@ -786,13 +786,16 @@ class BufferedWindowIcon( BufferedWindow ):
painter.setRenderHint( QG.QPainter.SmoothPixmapTransform, True ) # makes any scaling here due to jank thumbs look good
x_offset = ( self.width() - self._pixmap.width() ) / 2
y_offset = ( self.height() - self._pixmap.height() ) / 2
if isinstance( self._pixmap, QG.QImage ):
painter.drawImage( self.rect(), self._pixmap )
painter.drawImage( x_offset, y_offset, self._pixmap )
else:
painter.drawPixmap( self.rect(), self._pixmap )
painter.drawPixmap( x_offset, y_offset, self._pixmap )

4
hydrus/client/importing/ClientImportFiles.py
View File

@ -351,11 +351,11 @@ class FileImportJob( object ):
thumbnail_scale_type = HG.client_controller.new_options.GetInteger( 'thumbnail_scale_type' )
thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
( clip_rect, target_resolution ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( ( width, height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
target_resolution = HydrusImageHandling.GetThumbnailResolution( ( width, height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
percentage_in = HG.client_controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )
thumbnail_numpy = HydrusFileHandling.GenerateThumbnailNumPy(self._temp_path, target_resolution, mime, duration, num_frames, clip_rect = clip_rect, percentage_in = percentage_in)
thumbnail_numpy = HydrusFileHandling.GenerateThumbnailNumPy(self._temp_path, target_resolution, mime, duration, num_frames, percentage_in = percentage_in)
# this guy handles almost all his own exceptions now, so no need for clever catching. if it fails, we are prob talking an I/O failure, which is not a 'thumbnail failed' error
self._thumbnail_bytes = HydrusImageHandling.GenerateThumbnailBytesFromNumPy( thumbnail_numpy )

4
hydrus/client/importing/ClientImportSubscriptionLegacy.py
View File

@ -653,7 +653,7 @@ class SubscriptionLegacy( HydrusSerialisable.SerialisableBaseNamed ):
nj.engine = HG.client_controller.network_engine
if nj.NeedsLogin():
if nj.CurrentlyNeedsLogin():
try:
@ -713,7 +713,7 @@ class SubscriptionLegacy( HydrusSerialisable.SerialisableBaseNamed ):
nj.engine = HG.client_controller.network_engine
if nj.NeedsLogin():
if nj.CurrentlyNeedsLogin():
try:

4
hydrus/client/importing/ClientImportSubscriptionQuery.py
View File

@ -339,7 +339,7 @@ class SubscriptionQueryHeader( HydrusSerialisable.SerialisableBase ):
nj.engine = network_engine
if nj.NeedsLogin():
if nj.CurrentlyNeedsLogin():
try:
@ -389,7 +389,7 @@ class SubscriptionQueryHeader( HydrusSerialisable.SerialisableBase ):
nj.engine = network_engine
if nj.NeedsLogin():
if nj.CurrentlyNeedsLogin():
try:

2
hydrus/client/media/ClientMedia.py
View File

@ -1060,7 +1060,7 @@ class MediaList( object ):
if action == HC.CONTENT_UPDATE_DELETE:
local_file_domains = HG.client_controller.services_manager.GetServiceKeys( ( HC.LOCAL_FILE_DOMAIN, ) )
all_local_file_services = set( list( local_file_domains ) + [ CC.COMBINED_LOCAL_FILE_SERVICE_KEY, CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY, CC.TRASH_SERVICE_KEY ] )
all_local_file_services = set( list( local_file_domains ) + [ CC.COMBINED_LOCAL_FILE_SERVICE_KEY, CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY, CC.TRASH_SERVICE_KEY, CC.LOCAL_UPDATE_SERVICE_KEY ] )
#

10
hydrus/client/media/ClientMediaManagers.py
View File

@ -1140,6 +1140,16 @@ class LocationsManager( object ):
elif service_key == CC.COMBINED_LOCAL_FILE_SERVICE_KEY:
for s_k in HG.client_controller.services_manager.GetServiceKeys( ( HC.COMBINED_LOCAL_MEDIA, HC.LOCAL_FILE_DOMAIN, HC.LOCAL_FILE_TRASH_DOMAIN, HC.LOCAL_FILE_UPDATE_DOMAIN ) ):
if s_k in self._current:
self._DeleteFromService( s_k, reason )
else:
self._DeleteFromService( service_key, reason )

4
hydrus/client/networking/ClientLocalServerResources.py
View File

@ -3102,7 +3102,7 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
if width is not None and height is not None and width > 0 and height > 0:
( clip_rect, ( expected_thumbnail_width, expected_thumbnail_height ) ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( ( width, height ), thumbnail_bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
( expected_thumbnail_width, expected_thumbnail_height ) = HydrusImageHandling.GetThumbnailResolution( ( width, height ), thumbnail_bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
metadata_row[ 'thumbnail_width' ] = expected_thumbnail_width
metadata_row[ 'thumbnail_height' ] = expected_thumbnail_height
@ -3408,8 +3408,6 @@ class HydrusResourceClientAPIRestrictedManageCookiesGetCookies( HydrusResourceCl
body_cookies_list.append( [ name, value, domain, path, expires ] )
body_dict = {}
body_dict = { 'cookies' : body_cookies_list }
body = Dumps( body_dict, request.preferred_mime )

21
hydrus/client/networking/ClientNetworking.py
View File

@ -79,6 +79,13 @@ class NetworkEngine( object ):
self.controller.sub( self, 'RefreshOptions', 'notify_new_options' )
def _AssignCurrentLoginProcess( self, login_process: typing.Optional[ ClientNetworkingLogin.LoginProcess ] ):
self._current_login_process = login_process
self.login_manager.SetCurrentLoginProcess( login_process )
def AddJob( self, job: ClientNetworkingJobs.NetworkJob ):
if HG.network_report_mode:
@ -259,7 +266,7 @@ class NetworkEngine( object ):
self.controller.CallToThread( login_process.Start )
self._current_login_process = login_process
self._AssignCurrentLoginProcess( login_process )
@ -273,7 +280,7 @@ class NetworkEngine( object ):
return True
elif job.NeedsLogin():
elif job.CurrentlyNeedsLogin():
try:
@ -291,13 +298,15 @@ class NetworkEngine( object ):
else:
job_login_network_context = job.GetLoginNetworkContext()
if job.IsHydrusJob():
message = 'This hydrus service (' + job.GetLoginNetworkContext().ToString() + ') could not do work because: {}'.format( repr( e ) )
message = f'This hydrus service "{job_login_network_context.ToString()}" could not do work because: {e}'
else:
message = 'This job\'s network context (' + job.GetLoginNetworkContext().ToString() + ') seems to have an invalid login. The error was: {}'.format( repr( e ) )
message = f'This job\'s network context "{job_login_network_context.ToString()}" seems to have an invalid login. The error was: {e}'
job.Cancel( message )
@ -325,7 +334,7 @@ class NetworkEngine( object ):
self.controller.CallToThread( login_process.Start )
self._current_login_process = login_process
self._AssignCurrentLoginProcess( login_process )
job.SetStatus( 'logging in' + HC.UNICODE_ELLIPSIS )
@ -350,7 +359,7 @@ class NetworkEngine( object ):
if self._current_login_process.IsDone():
self._current_login_process = None
self._AssignCurrentLoginProcess( None )

30
hydrus/client/networking/ClientNetworkingJobs.py
View File

@ -1137,6 +1137,21 @@ class NetworkJob( object ):
def CurrentlyNeedsLogin( self ):
with self._lock:
if self._for_login:
return False
else:
return self.engine.login_manager.CurrentlyNeedsLogin( self._login_network_context )
def CurrentlyWaitingOnConnectionError( self ):
with self._lock:
@ -1388,21 +1403,6 @@ class NetworkJob( object ):
def NeedsLogin( self ):
with self._lock:
if self._for_login:
return False
else:
return self.engine.login_manager.NeedsLogin( self._login_network_context )
def NoEngineYet( self ):
return self.engine is None

99
hydrus/client/networking/ClientNetworkingLogin.py
View File

@ -3,6 +3,7 @@ import os
import re
import threading
import time
import typing
import urllib.parse
from hydrus.core import HydrusGlobals as HG
@ -75,6 +76,8 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
self._login_script_keys_to_login_scripts = {}
self._login_script_names_to_login_scripts = {}
self._current_login_process: typing.Optional[ LoginProcess ] = None
self._hydrus_login_script = LoginScriptHydrus()
self._error_names = set()
@ -324,11 +327,11 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
if login_domain is None or not login_expected:
raise HydrusExceptions.ValidationException( 'The domain ' + login_domain + ' has no active login script--has it just been turned off?' )
raise HydrusExceptions.ValidationException( f'The domain "{login_domain}" has no active login script--has it just been turned off?' )
elif not login_possible:
raise HydrusExceptions.ValidationException( 'The domain ' + login_domain + ' cannot log in: ' + login_error_text )
raise HydrusExceptions.ValidationException( f'The domain "{login_domain}" cannot log in: {login_error_text}' )
elif network_context.context_type == CC.NETWORK_CONTEXT_HYDRUS:
@ -358,6 +361,50 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
def CurrentlyNeedsLogin( self, network_context ):
with self._lock:
if self._current_login_process is not None and self._current_login_process.network_context == network_context:
# this network context is currently being logged in, so yes, we still need to wait for that to finish
return True
if network_context.context_type == CC.NETWORK_CONTEXT_DOMAIN:
( login_domain, login_expected, login_possible, login_error_text ) = self._GetLoginDomainStatus( network_context )
if login_domain is None or not login_expected:
return False # no login required, no problem
else:
try:
( login_script, credentials ) = self._GetLoginScriptAndCredentials( login_domain )
except HydrusExceptions.ValidationException:
# couldn't find the script or something. assume we need a login to move errors forward to checkcanlogin trigger phase
return True
login_network_context = ClientNetworkingContexts.NetworkContext( context_type = CC.NETWORK_CONTEXT_DOMAIN, context_data = login_domain )
return not login_script.IsLoggedIn( self.engine, login_network_context )
elif network_context.context_type == CC.NETWORK_CONTEXT_HYDRUS:
return not self._hydrus_login_script.IsLoggedIn( self.engine, network_context )
def DelayLoginScript( self, login_domain, login_script_key, reason ):
with self._lock:
@ -529,43 +576,6 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
def NeedsLogin( self, network_context ):
with self._lock:
if network_context.context_type == CC.NETWORK_CONTEXT_DOMAIN:
( login_domain, login_expected, login_possible, login_error_text ) = self._GetLoginDomainStatus( network_context )
if login_domain is None or not login_expected:
return False # no login required, no problem
else:
try:
( login_script, credentials ) = self._GetLoginScriptAndCredentials( login_domain )
except HydrusExceptions.ValidationException:
# couldn't find the script or something. assume we need a login to move errors forward to checkcanlogin trigger phase
return True
login_network_context = ClientNetworkingContexts.NetworkContext( context_type = CC.NETWORK_CONTEXT_DOMAIN, context_data = login_domain )
return not login_script.IsLoggedIn( self.engine, login_network_context )
elif network_context.context_type == CC.NETWORK_CONTEXT_HYDRUS:
return not self._hydrus_login_script.IsLoggedIn( self.engine, network_context )
def OverwriteDefaultLoginScripts( self, login_script_names ):
with self._lock:
@ -619,6 +629,12 @@ class NetworkLoginManager( HydrusSerialisable.SerialisableBase ):
def SetCurrentLoginProcess( self, login_process: typing.Optional[ "LoginProcess" ] ):
self._current_login_process = login_process
def SetDomainsToLoginInfo( self, domains_to_login_info ):
with self._lock:
@ -904,6 +920,11 @@ class LoginProcess( object ):
raise NotImplementedError()
def GetNetworkContext( self ):
return self.network_context
def IsDone( self ):
return self._done

101
hydrus/client/networking/ClientNetworkingSessions.py
View File

@ -29,6 +29,9 @@ class NetworkSessionManagerSessionContainer( HydrusSerialisable.SerialisableBase
SERIALISABLE_NAME = 'Session Manager Session Container'
SERIALISABLE_VERSION = 2
POOL_CONNECTION_TIMEOUT = 5 * 60
SESSION_TIMEOUT = 45 * 60
def __init__( self, name, network_context = None, session = None ):
if network_context is None:
@ -40,6 +43,10 @@ class NetworkSessionManagerSessionContainer( HydrusSerialisable.SerialisableBase
self.network_context = network_context
self.session = session
self.last_touched_time = HydrusTime.GetNow()
self.pool_is_cleared = True
self.printed_connection_pool_error = False
def _InitialiseEmptySession( self ):
@ -108,6 +115,55 @@ class NetworkSessionManagerSessionContainer( HydrusSerialisable.SerialisableBase
def MaintainConnectionPool( self ):
if not self.pool_is_cleared and HydrusTime.TimeHasPassed( self.last_touched_time + self.POOL_CONNECTION_TIMEOUT ):
try:
my_session_adapters = list( self.session.adapters.values() )
for adapter in my_session_adapters:
poolmanager = getattr( adapter, 'poolmanager', None )
if poolmanager is not None:
poolmanager.clear()
self.pool_is_cleared = True
except Exception as e:
if not self.printed_connection_pool_error:
self.printed_connection_pool_error = True
HydrusData.Print( 'There was a problem clearing the connection pool, this message will not be printed again this boot:' )
HydrusData.PrintException( e, do_wait = False )
def PrepareForNewWork( self ):
self.last_touched_time = HydrusTime.GetNow()
self.pool_is_cleared = False
my_session_cookies = self.session.cookies
if HydrusTime.TimeHasPassed( self.last_touched_time + self.SESSION_TIMEOUT ):
my_session_cookies.clear_session_cookies()
my_session_cookies.clear_expired_cookies()
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER_SESSION_CONTAINER ] = NetworkSessionManagerSessionContainer
class NetworkSessionManager( HydrusSerialisable.SerialisableBase ):
@ -116,8 +172,6 @@ class NetworkSessionManager( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_NAME = 'Session Manager'
SERIALISABLE_VERSION = 1
SESSION_TIMEOUT = 45 * 60
def __init__( self ):
HydrusSerialisable.SerialisableBase.__init__( self )
@ -133,30 +187,12 @@ class NetworkSessionManager( HydrusSerialisable.SerialisableBase ):
self._session_container_names_to_session_containers = {}
self._network_contexts_to_session_containers = {}
self._network_contexts_to_session_timeouts = {}
self._proxies_dict = {}
self._ReinitialiseProxies()
HG.client_controller.sub( self, 'ReinitialiseProxies', 'notify_new_options' )
def _CleanSessionCookies( self, network_context, session ):
if network_context not in self._network_contexts_to_session_timeouts:
self._network_contexts_to_session_timeouts[ network_context ] = 0
if HydrusTime.TimeHasPassed( self._network_contexts_to_session_timeouts[ network_context ] ):
session.cookies.clear_session_cookies()
self._network_contexts_to_session_timeouts[ network_context ] = HydrusTime.GetNow() + self.SESSION_TIMEOUT
session.cookies.clear_expired_cookies()
HG.client_controller.sub( self, 'MaintainConnectionPools', 'memory_maintenance_pulse' )
def _GetSerialisableInfo( self ):
@ -239,11 +275,6 @@ class NetworkSessionManager( HydrusSerialisable.SerialisableBase ):
network_context = self._GetSessionNetworkContext( network_context )
if network_context in self._network_contexts_to_session_timeouts:
del self._network_contexts_to_session_timeouts[ network_context ]
if network_context in self._network_contexts_to_session_containers:
session_container = self._network_contexts_to_session_containers[ network_context ]
@ -301,17 +332,17 @@ class NetworkSessionManager( HydrusSerialisable.SerialisableBase ):
self._InitialiseSessionContainer( network_context )
session = self._network_contexts_to_session_containers[ network_context ].session
session_container = self._network_contexts_to_session_containers[ network_context ]
session_container.PrepareForNewWork()
session = session_container.session
if session.proxies != self._proxies_dict:
session.proxies = dict( self._proxies_dict )
#
self._CleanSessionCookies( network_context, session )
#
# tumblr can't into ssl for some reason, and the data subdomain they use has weird cert properties, looking like amazon S3
@ -354,6 +385,14 @@ class NetworkSessionManager( HydrusSerialisable.SerialisableBase ):
def MaintainConnectionPools( self ):
for session_container in self._network_contexts_to_session_containers.values():
session_container.MaintainConnectionPool()
def ReinitialiseProxies( self ):
with self._lock:

2
hydrus/core/HydrusConstants.py
View File

@ -103,7 +103,7 @@ options = {}
# Misc
NETWORK_VERSION = 20
SOFTWARE_VERSION = 549
SOFTWARE_VERSION = 550
CLIENT_API_VERSION = 54
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )

50
hydrus/core/HydrusFileHandling.py
View File

@ -42,14 +42,14 @@ except Exception as e:
def GenerateThumbnailBytes( path, target_resolution, mime, duration, num_frames, clip_rect = None, percentage_in = 35 ):
def GenerateThumbnailBytes( path, target_resolution, mime, duration, num_frames, percentage_in = 35 ):
thumbnail_numpy = GenerateThumbnailNumPy(path, target_resolution, mime, duration, num_frames, clip_rect, percentage_in )
thumbnail_numpy = GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames, percentage_in = percentage_in )
return HydrusImageHandling.GenerateThumbnailBytesFromNumPy( thumbnail_numpy )
def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames, clip_rect = None, percentage_in = 35 ):
def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames, percentage_in = 35 ):
if target_resolution == ( 0, 0 ):
@ -60,7 +60,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
try:
thumbnail_numpy = HydrusPSDHandling.GenerateThumbnailNumPyFromPSDPath( path, target_resolution, clip_rect = clip_rect )
thumbnail_numpy = HydrusPSDHandling.GenerateThumbnailNumPyFromPSDPath( path, target_resolution )
except Exception as e:
@ -68,19 +68,19 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
HydrusData.PrintException( e )
HydrusData.Print( 'Attempting ffmpeg PSD thumbnail fallback' )
( os_file_handle, temp_path ) = HydrusTemp.GetTempPath( suffix = '.png' )
try:
( os_file_handle, temp_path ) = HydrusTemp.GetTempPath( suffix = '.png' )
HydrusVideoHandling.RenderImageToImagePath( path, temp_path )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( temp_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( temp_path, target_resolution, HC.IMAGE_PNG )
except Exception as e:
thumb_path = os.path.join( HC.STATIC_DIR, 'psd.png' )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG )
finally:
@ -96,13 +96,13 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
HydrusClipHandling.ExtractDBPNGToPath( path, temp_path )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( temp_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( temp_path, target_resolution, HC.IMAGE_PNG )
except:
thumb_path = os.path.join( HC.STATIC_DIR, 'clip.png' )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG )
finally:
@ -113,7 +113,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
try:
thumbnail_numpy = HydrusKritaHandling.GenerateThumbnailNumPyFromKraPath( path, target_resolution, clip_rect = clip_rect )
thumbnail_numpy = HydrusKritaHandling.GenerateThumbnailNumPyFromKraPath( path, target_resolution )
except Exception as e:
@ -125,7 +125,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
thumb_path = os.path.join( HC.STATIC_DIR, 'krita.png' )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG )
elif mime == HC.APPLICATION_PROCREATE:
@ -136,13 +136,13 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
HydrusProcreateHandling.ExtractZippedThumbnailToPath( path, temp_path )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( temp_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( temp_path, target_resolution, HC.IMAGE_PNG )
except Exception as e:
thumb_path = os.path.join( HC.STATIC_DIR, 'procreate.png' )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG )
finally:
@ -153,7 +153,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
try:
thumbnail_numpy = HydrusSVGHandling.GenerateThumbnailNumPyFromSVGPath( path, target_resolution, clip_rect = clip_rect )
thumbnail_numpy = HydrusSVGHandling.GenerateThumbnailNumPyFromSVGPath( path, target_resolution )
except Exception as e:
@ -165,14 +165,14 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
thumb_path = os.path.join( HC.STATIC_DIR, 'svg.png' )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG )
elif mime == HC.APPLICATION_PDF:
try:
thumbnail_numpy = HydrusPDFHandling.GenerateThumbnailNumPyFromPDFPath( path, target_resolution, clip_rect = clip_rect )
thumbnail_numpy = HydrusPDFHandling.GenerateThumbnailNumPyFromPDFPath( path, target_resolution )
except Exception as e:
@ -184,7 +184,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
thumb_path = os.path.join( HC.STATIC_DIR, 'pdf.png' )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG )
elif mime == HC.APPLICATION_FLASH:
@ -195,13 +195,13 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
HydrusFlashHandling.RenderPageToFile( path, temp_path, 1 )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( temp_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( temp_path, target_resolution, HC.IMAGE_PNG )
except:
thumb_path = os.path.join( HC.STATIC_DIR, 'flash.png' )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG )
finally:
@ -214,7 +214,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
try:
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( path, target_resolution, mime, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( path, target_resolution, mime )
except Exception as e:
@ -223,7 +223,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
thumb_path = os.path.join( HC.STATIC_DIR, 'hydrus.png' )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG )
else:
@ -234,7 +234,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
try:
renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, target_resolution, clip_rect = clip_rect, start_pos = desired_thumb_frame )
renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, target_resolution, start_pos = desired_thumb_frame )
numpy_image = renderer.read_frame()
@ -257,7 +257,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
try:
renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, target_resolution, clip_rect = clip_rect )
renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, target_resolution )
numpy_image = renderer.read_frame()
@ -274,7 +274,7 @@ def GenerateThumbnailNumPy( path, target_resolution, mime, duration, num_frames,
thumb_path = os.path.join( HC.STATIC_DIR, 'hydrus.png' )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG, clip_rect = clip_rect )
thumbnail_numpy = HydrusImageHandling.GenerateThumbnailNumPyFromStaticImagePath( thumb_path, target_resolution, HC.IMAGE_PNG )
else:

7
hydrus/core/HydrusKritaHandling.py
View File

@ -39,7 +39,7 @@ def ThumbnailPILImageFromKra(path):
def GenerateThumbnailNumPyFromKraPath( path: str, target_resolution: typing.Tuple[int, int], clip_rect = None ) -> bytes:
def GenerateThumbnailNumPyFromKraPath( path: str, target_resolution: typing.Tuple[ int, int ] ) -> bytes:
try:
@ -50,11 +50,6 @@ def GenerateThumbnailNumPyFromKraPath( path: str, target_resolution: typing.Tupl
pil_image = ThumbnailPILImageFromKra( path )
if clip_rect is not None:
pil_image = HydrusImageHandling.ClipPILImage( pil_image, clip_rect )
thumbnail_pil_image = pil_image.resize( target_resolution, PILImage.LANCZOS )
numpy_image = HydrusImageHandling.GenerateNumPyImageFromPILImage( thumbnail_pil_image )

2
hydrus/core/HydrusPDFHandling.py
View File

@ -2,7 +2,7 @@ import typing
from hydrus.core import HydrusExceptions
def BaseGenerateThumbnailNumPyFromPDFPath( path: str, target_resolution: typing.Tuple[int, int], clip_rect = None ) -> bytes:
def BaseGenerateThumbnailNumPyFromPDFPath( path: str, target_resolution: typing.Tuple[int, int] ) -> bytes:
raise HydrusExceptions.NoThumbnailFileException()

7
hydrus/core/HydrusPSDHandling.py
View File

@ -39,15 +39,10 @@ def MergedPILImageFromPSD( path: str ) -> PILImage:
return pil_image
def GenerateThumbnailNumPyFromPSDPath( path: str, target_resolution: typing.Tuple[int, int], clip_rect = None ) -> bytes:
def GenerateThumbnailNumPyFromPSDPath( path: str, target_resolution: typing.Tuple[int, int] ) -> bytes:
pil_image = MergedPILImageFromPSD( path )
if clip_rect is not None:
pil_image = HydrusImageHandling.ClipPILImage( pil_image, clip_rect )
thumbnail_pil_image = pil_image.resize( target_resolution, PILImage.LANCZOS )
numpy_image = HydrusImageHandling.GenerateNumPyImageFromPILImage( thumbnail_pil_image )

2
hydrus/core/HydrusSVGHandling.py
View File

@ -2,7 +2,7 @@ import typing
from hydrus.core import HydrusExceptions
def BaseGenerateThumbnailNumPyFromSVGPath( path: str, target_resolution: typing.Tuple[int, int], clip_rect = None ) -> bytes:
def BaseGenerateThumbnailNumPyFromSVGPath( path: str, target_resolution: typing.Tuple[int, int] ) -> bytes:
raise HydrusExceptions.NoThumbnailFileException()

90
hydrus/core/images/HydrusImageHandling.py
View File

@ -338,15 +338,10 @@ def GeneratePILImageFromNumPyImage( numpy_image: numpy.array ) -> PILImage.Image
return pil_image
def GenerateThumbnailNumPyFromStaticImagePath( path, target_resolution, mime, clip_rect = None ):
def GenerateThumbnailNumPyFromStaticImagePath( path, target_resolution, mime ):
numpy_image = GenerateNumPyImage( path, mime )
if clip_rect is not None:
numpy_image = ClipNumPyImage( numpy_image, clip_rect )
thumbnail_numpy_image = ResizeNumPyImage( numpy_image, target_resolution )
return thumbnail_numpy_image
@ -517,9 +512,7 @@ thumbnail_scale_str_lookup = {
THUMBNAIL_SCALE_TO_FILL : 'scale to fill'
}
def GetThumbnailResolutionAndClipRegion( image_resolution: typing.Tuple[ int, int ], bounding_dimensions: typing.Tuple[ int, int ], thumbnail_scale_type: int, thumbnail_dpr_percent: int ):
clip_rect = None
def GetThumbnailResolution( image_resolution: typing.Tuple[ int, int ], bounding_dimensions: typing.Tuple[ int, int ], thumbnail_scale_type: int, thumbnail_dpr_percent: int ) -> typing.Tuple[ int, int ]:
( im_width, im_height ) = image_resolution
( bounding_width, bounding_height ) = bounding_dimensions
@ -530,7 +523,8 @@ def GetThumbnailResolutionAndClipRegion( image_resolution: typing.Tuple[ int, in
bounding_height = int( bounding_height * thumbnail_dpr )
bounding_width = int( bounding_width * thumbnail_dpr )
if im_width is None:
im_width = bounding_width
@ -546,61 +540,75 @@ def GetThumbnailResolutionAndClipRegion( image_resolution: typing.Tuple[ int, in
if bounding_width >= im_width and bounding_height >= im_height:
return ( clip_rect, ( im_width, im_height ) )
return ( im_width, im_height )
image_ratio = im_width / im_height
width_ratio = im_width / bounding_width
height_ratio = im_height / bounding_height
image_is_wider_than_bounding_box = width_ratio > height_ratio
image_is_taller_than_bounding_box = height_ratio > width_ratio
thumbnail_width = bounding_width
thumbnail_height = bounding_height
if thumbnail_scale_type in ( THUMBNAIL_SCALE_DOWN_ONLY, THUMBNAIL_SCALE_TO_FIT ):
if width_ratio > height_ratio:
thumbnail_height = im_height / width_ratio
elif height_ratio > width_ratio:
if image_is_taller_than_bounding_box: # i.e. the height will be at bounding height
thumbnail_width = im_width / height_ratio
elif image_is_wider_than_bounding_box: # i.e. the width will be at bounding width
thumbnail_height = im_height / width_ratio
elif thumbnail_scale_type == THUMBNAIL_SCALE_TO_FILL:
if width_ratio == height_ratio:
# we do min 5.0 here to stop really tall and thin images getting zoomed in from width 1px to 150 and getting a thumbnail with a height of 75,000 pixels
# in this case the line image is already crazy distorted, so we don't mind squishing it
if image_is_taller_than_bounding_box: # i.e. the width will be at bounding width, the height will spill over
# we have something that fits bounding region perfectly, no clip region required
thumbnail_height = bounding_width * min( 5.0, 1 / image_ratio )
pass
elif image_is_wider_than_bounding_box: # i.e. the height will be at bounding height, the width will spill over
else:
clip_x = 0
clip_y = 0
clip_width = im_width
clip_height = im_height
if width_ratio > height_ratio:
clip_width = max( int( im_width * height_ratio / width_ratio ), 1 )
clip_x = ( im_width - clip_width ) // 2
elif height_ratio > width_ratio:
clip_height = max( int( im_height * width_ratio / height_ratio ), 1 )
clip_y = ( im_height - clip_height ) // 2
clip_rect = ( clip_x, clip_y, clip_width, clip_height )
thumbnail_width = bounding_height * min( 5.0, image_ratio )
# old stuff that actually clipped the size of the thing
'''
clip_x = 0
clip_y = 0
clip_width = im_width
clip_height = im_height
if width_ratio > height_ratio:
clip_width = max( int( im_width * height_ratio / width_ratio ), 1 )
clip_x = ( im_width - clip_width ) // 2
elif height_ratio > width_ratio:
clip_height = max( int( im_height * width_ratio / height_ratio ), 1 )
clip_y = ( im_height - clip_height ) // 2
clip_rect = ( clip_x, clip_y, clip_width, clip_height )
'''
thumbnail_width = max( int( thumbnail_width ), 1 )
thumbnail_height = max( int( thumbnail_height ), 1 )
thumbnail_width = int( thumbnail_width )
thumbnail_height = int( thumbnail_height )
return ( clip_rect, ( thumbnail_width, thumbnail_height ) )
thumbnail_width = max( thumbnail_width, 1 )
thumbnail_height = max( thumbnail_height, 1 )
return ( thumbnail_width, thumbnail_height )
def IsDecompressionBomb( path ) -> bool:

4
hydrus/core/images/HydrusImageNormalisation.py
View File

@ -88,7 +88,7 @@ def NormaliseICCProfilePILImageToSRGB( pil_image: PILImage.Image ) -> PILImage.I
src_profile = PILImageCms.ImageCmsProfile( f )
if pil_image.mode in ( 'L', 'LA' ):
if pil_image.mode in ( 'L', 'LA', 'P' ):
# had a bunch of LA pngs that turned pure white on RGBA ICC conversion
# but seem to work fine if keep colourspace the same for now
@ -123,8 +123,6 @@ def NormaliseICCProfilePILImageToSRGB( pil_image: PILImage.Image ) -> PILImage.I
pass
pil_image = NormalisePILImageToRGB( pil_image )
return pil_image

4
hydrus/core/networking/HydrusNetworkVariableHandling.py
View File

@ -190,9 +190,9 @@ def ParseFileArguments( path, decompression_bombs_ok = False ):
bounding_dimensions = HC.SERVER_THUMBNAIL_DIMENSIONS
( clip_rect, target_resolution ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( ( width, height ), bounding_dimensions, HydrusImageHandling.THUMBNAIL_SCALE_DOWN_ONLY, 100 )
target_resolution = HydrusImageHandling.GetThumbnailResolution( ( width, height ), bounding_dimensions, HydrusImageHandling.THUMBNAIL_SCALE_DOWN_ONLY, 100 )
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( path, target_resolution, mime, duration, num_frames, clip_rect = clip_rect )
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( path, target_resolution, mime, duration, num_frames )
except Exception as e:

2
hydrus/test/TestClientAPI.py
View File

@ -4883,7 +4883,7 @@ class TestClientAPI( unittest.TestCase ):
thumbnail_scale_type = HG.test_controller.new_options.GetInteger( 'thumbnail_scale_type' )
thumbnail_dpr_percent = HG.client_controller.new_options.GetInteger( 'thumbnail_dpr_percent' )
( clip_rect, ( thumbnail_expected_width, thumbnail_expected_height ) ) = HydrusImageHandling.GetThumbnailResolutionAndClipRegion( ( file_info_manager.width, file_info_manager.height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
( thumbnail_expected_width, thumbnail_expected_height ) = HydrusImageHandling.GetThumbnailResolution( ( file_info_manager.width, file_info_manager.height ), bounding_dimensions, thumbnail_scale_type, thumbnail_dpr_percent )
metadata_row[ 'thumbnail_width' ] = thumbnail_expected_width
metadata_row[ 'thumbnail_height' ] = thumbnail_expected_height

1
static/build_files/windows/hydrus_client.spec
View File

@ -21,6 +21,7 @@ a = Analysis(['hydrus\\hydrus_client.pyw'],
('hydrus\\static', 'static'),
('hydrus\\license.txt', '.'),
('hydrus\\README.md', '.'),
('hydrus\\auto_update_installer.bat', '.'),
('hydrus\\help my client will not boot.txt', '.'),
('hydrus\\db', 'db'),
(cloudscraper_dir, 'cloudscraper')