RepoMirrors
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Version 236

This commit is contained in:
Hydrus Network Developer 2016-12-14 15:19:07 -06:00
parent 0657fe6e7f
commit 3859d7ff83
32 changed files with 1768 additions and 1084 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
help/changelog.html
View File

@ -8,6 +8,36 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 236</h3></li>
<ul>
<li>in prep for a network https upgrade, the client can now detect and escalate to https when making connections to hydrus services</li>
<li>import/export to png and clipboard now supports multiple objects at once!</li>
<li>rewrote the manage subscriptions dialog to work on the new panel system</li>
<li>the new manage subscriptions dialog has a listctrl and a sub edit dialog</li>
<li>the new manage subscriptions dialog has the same add/export/import/dupe/edit/delete buttons as the manage scripts dialog</li>
<li>subscriptions are now importable/exportable, including en masse with the new multiple object import/export support!</li>
<li>the new manage subscriptions dialog has retry failed/pause-resume/check now/reset buttons for easy mass subs management</li>
<li>the edit subscription panel has a bit of a layout makeover</li>
<li>the edit subscription panel now updates itself as its buttons are hit</li>
<li>the edit subscription panel disables buttons that are not applicable</li>
<li>subscriptions can now be renamed!</li>
<li>cleaned some misc subscription code</li>
<li>relabelled initial and periodic file limit in the subscription edit panel</li>
<li>middle-clicking on the main gui's greyspace (e.g. to the right of the notebook tabs) will spawn the new page chooser!</li>
<li>created a simple HydrusRatingArchive class--will do more with it in future</li>
<li>added ffmpeg, python, and sqlite versions to the help->about window</li>
<li>harmonised daemon code</li>
<li>added a new class of daemon that will not fire while a session load is occuring</li>
<li>subscriptions, import and export folders, and file repo downloads now use this new daemon</li>
<li>cleaned the way background daemons check for idle</li>
<li>expand/collapse panels now notify the new kind of toplevelwindow that a resize may be needed when they switch state</li>
<li>time deltas (like on subs edit panel or a thread watcher) now render more concisely ('7 days' instead of '7 days 0 hours')</li>
<li>serialisable object png export panel now has a width parameter</li>
<li>fixed a bug where tags that begin with unicode digits were accidentally identifying as numbers for the purposes of sorting and throwing errors on convert fail</li>
<li>the media viewer can handle some more unusual content update combinations--for instance, if it cannot figure out which media to show next, it will revert back to the first image rather than displaying an undefined null mess</li>
<li>updated and cleaned a bunch of my old misc encryption code</li>
<li>misc cleanup</li>
</ul>
<li><h3>version 235</h3></li>
<ul>
<li>finished first version of new faster dupe search--'system:similar to' now uses it</li>

2
help/running_from_source.html
View File

@ -11,7 +11,7 @@
<h3>what you will need</h3>
<p>You will need to install python 2.7 and a number of python modules. Most of it you can get through pip. I think this will do for most systems:</p>
<ul>
<li>pip install beautifulsoup4 hsaudiotag lxml lz4 nose numpy pafy Pillow psutil pycrypto PyPDF2 PySocks python-potr PyYAML requests Send2Trash twisted</li>
<li>pip install beautifulsoup4 hsaudiotag lxml lz4 nose numpy pafy Pillow psutil pycrypto PyOpenSSL PyPDF2 PySocks python-potr PyYAML requests Send2Trash service_identity twisted</li>
</ul>
<p>Although you may want to do them one at a time, or in a different order. Your specific system may also need some of them from different sources and will need some complicated things installed separately. The best way to figure it out is just to keep running client.pyw and see what it complains about missing.</p>
<p>I use Ubuntu 16.10, which also requires something like:</p>

22
include/ClientConstants.py
View File

@ -1,10 +1,7 @@
import collections
import cv2
import HydrusConstants as HC
import HydrusExceptions
import os
import PIL
import ssl
import sys
import threading
import traceback
@ -43,23 +40,6 @@ Shift-LeftClick-Drag - Drag (in Filter)
Ctrl + MouseWheel - Zoom
Z - Zoom Full/Fit'''
library_versions = []
library_versions.append( ( 'openssl', ssl.OPENSSL_VERSION ) )
library_versions.append( ( 'PIL', PIL.VERSION ) )
if hasattr( PIL, 'PILLOW_VERSION' ):
library_versions.append( ( 'Pillow', PIL.PILLOW_VERSION ) )
library_versions.append( ( 'OpenCV', cv2.__version__ ) )
library_versions.append( ( 'wx', wx.version() ) )
CLIENT_DESCRIPTION = 'This client is the media management application of the hydrus software suite.'
CLIENT_DESCRIPTION += os.linesep * 2 + os.linesep.join( ( lib + ': ' + version for ( lib, version ) in library_versions ) )
COLLECT_BY_S = 0
COLLECT_BY_SV = 1
COLLECT_BY_SVC = 2
@ -478,6 +458,8 @@ LOCAL_BOORU_SERVICE_KEY = 'local booru'
TRASH_SERVICE_KEY = 'trash'
COMBINED_LOCAL_FILES_SERVICE_KEY = 'all local files'
COMBINED_FILE_SERVICE_KEY = 'all known files'
COMBINED_TAG_SERVICE_KEY = 'all known tags'

30
include/ClientController.py
View File

@ -196,15 +196,17 @@ class Controller( HydrusController.HydrusController ):
elif mouse_position != self._last_mouse_position:
idle_before = self.CurrentlyIdle()
idle_before_position_update = self.CurrentlyIdle()
self._timestamps[ 'last_mouse_action' ] = HydrusData.GetNow()
self._last_mouse_position = mouse_position
idle_after = self.CurrentlyIdle()
idle_after_position_update = self.CurrentlyIdle()
if idle_before != idle_after:
move_knocked_us_out_of_idle = ( not idle_before_position_update ) and idle_after_position_update
if move_knocked_us_out_of_idle:
self.pub( 'refresh_status' )
@ -509,6 +511,11 @@ class Controller( HydrusController.HydrusController ):
return self._db.GetUpdatesDir()
def GoodTimeToDoForegroundWork( self ):
return not self._gui.CurrentlyBusy()
def InitModel( self ):
self.pub( 'splash_set_title_text', 'booting db...' )
@ -623,16 +630,17 @@ class Controller( HydrusController.HydrusController ):
if not self._no_daemons:
self._daemons.append( HydrusThreading.DAEMONWorker( self, 'CheckMouseIdle', ClientDaemons.DAEMONCheckMouseIdle, period = 10 ) )
self._daemons.append( HydrusThreading.DAEMONWorker( self, 'DownloadFiles', ClientDaemons.DAEMONDownloadFiles, ( 'notify_new_downloads', 'notify_new_permissions' ) ) )
self._daemons.append( HydrusThreading.DAEMONWorker( self, 'SynchroniseAccounts', ClientDaemons.DAEMONSynchroniseAccounts, ( 'permissions_are_stale', ) ) )
self._daemons.append( HydrusThreading.DAEMONWorker( self, 'SynchroniseSubscriptions', ClientDaemons.DAEMONSynchroniseSubscriptions, ( 'notify_restart_subs_sync_daemon', 'notify_new_subscriptions' ), init_wait = 90 ) )
self._daemons.append( HydrusThreading.DAEMONBigJobWorker( self, 'CheckImportFolders', ClientDaemons.DAEMONCheckImportFolders, ( 'notify_restart_import_folders_daemon', 'notify_new_import_folders' ), period = 180 ) )
self._daemons.append( HydrusThreading.DAEMONBigJobWorker( self, 'CheckExportFolders', ClientDaemons.DAEMONCheckExportFolders, ( 'notify_restart_export_folders_daemon', 'notify_new_export_folders' ), period = 180 ) )
self._daemons.append( HydrusThreading.DAEMONBigJobWorker( self, 'MaintainTrash', ClientDaemons.DAEMONMaintainTrash, init_wait = 60 ) )
self._daemons.append( HydrusThreading.DAEMONBigJobWorker( self, 'RebalanceClientFiles', ClientDaemons.DAEMONRebalanceClientFiles, period = 3600 ) )
self._daemons.append( HydrusThreading.DAEMONBigJobWorker( self, 'SynchroniseRepositories', ClientDaemons.DAEMONSynchroniseRepositories, ( 'notify_restart_repo_sync_daemon', 'notify_new_permissions' ), period = 4 * 3600 ) )
self._daemons.append( HydrusThreading.DAEMONBigJobWorker( self, 'UPnP', ClientDaemons.DAEMONUPnP, ( 'notify_new_upnp_mappings', ), init_wait = 120, pre_callable_wait = 6 ) )
self._daemons.append( HydrusThreading.DAEMONForegroundWorker( self, 'DownloadFiles', ClientDaemons.DAEMONDownloadFiles, ( 'notify_new_downloads', 'notify_new_permissions' ) ) )
self._daemons.append( HydrusThreading.DAEMONForegroundWorker( self, 'SynchroniseSubscriptions', ClientDaemons.DAEMONSynchroniseSubscriptions, ( 'notify_restart_subs_sync_daemon', 'notify_new_subscriptions' ), init_wait = 60, pre_call_wait = 3 ) )
self._daemons.append( HydrusThreading.DAEMONForegroundWorker( self, 'CheckImportFolders', ClientDaemons.DAEMONCheckImportFolders, ( 'notify_restart_import_folders_daemon', 'notify_new_import_folders' ), period = 180 ) )
self._daemons.append( HydrusThreading.DAEMONForegroundWorker( self, 'CheckExportFolders', ClientDaemons.DAEMONCheckExportFolders, ( 'notify_restart_export_folders_daemon', 'notify_new_export_folders' ), period = 180 ) )
self._daemons.append( HydrusThreading.DAEMONBackgroundWorker( self, 'MaintainTrash', ClientDaemons.DAEMONMaintainTrash, init_wait = 60 ) )
self._daemons.append( HydrusThreading.DAEMONBackgroundWorker( self, 'RebalanceClientFiles', ClientDaemons.DAEMONRebalanceClientFiles, period = 3600 ) )
self._daemons.append( HydrusThreading.DAEMONBackgroundWorker( self, 'SynchroniseRepositories', ClientDaemons.DAEMONSynchroniseRepositories, ( 'notify_restart_repo_sync_daemon', 'notify_new_permissions' ), period = 4 * 3600, pre_call_wait = 3 ) )
self._daemons.append( HydrusThreading.DAEMONBackgroundWorker( self, 'UPnP', ClientDaemons.DAEMONUPnP, ( 'notify_new_upnp_mappings', ), init_wait = 120, pre_call_wait = 6 ) )
self._daemons.append( HydrusThreading.DAEMONQueue( self, 'FlushRepositoryUpdates', ClientDaemons.DAEMONFlushServiceUpdates, 'service_updates_delayed', period = 5 ) )

12
include/ClientDaemons.py
View File

@ -243,10 +243,7 @@ def DAEMONMaintainTrash( controller ):
def DAEMONRebalanceClientFiles( controller ):
if controller.CurrentlyIdle():
controller.GetClientFilesManager().Rebalance()
controller.GetClientFilesManager().Rebalance()
def DAEMONSynchroniseAccounts( controller ):
@ -322,10 +319,7 @@ def DAEMONSynchroniseRepositories( controller ):
break
if controller.CurrentlyIdle():
service.Sync( only_when_idle = True )
service.Sync( only_when_idle = True )
time.sleep( 5 )
@ -422,4 +416,4 @@ def DAEMONUPnP( controller ):

3
include/ClientData.py
View File

@ -605,6 +605,7 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
self._dictionary[ 'frame_locations' ][ 'file_import_status' ] = ( True, True, None, None, ( -1, -1 ), 'topleft', False, False )
self._dictionary[ 'frame_locations' ][ 'main_gui' ] = ( True, True, ( 640, 480 ), ( 20, 20 ), ( -1, -1 ), 'topleft', True, False )
self._dictionary[ 'frame_locations' ][ 'manage_options_dialog' ] = ( False, False, None, None, ( -1, -1 ), 'topleft', False, False )
self._dictionary[ 'frame_locations' ][ 'manage_subscriptions_dialog' ] = ( True, True, None, None, ( 1, -1 ), 'topleft', False, False )
self._dictionary[ 'frame_locations' ][ 'manage_tags_dialog' ] = ( False, False, None, None, ( -1, 1 ), 'topleft', False, False )
self._dictionary[ 'frame_locations' ][ 'manage_tags_frame' ] = ( False, False, None, None, ( -1, 1 ), 'topleft', False, False )
self._dictionary[ 'frame_locations' ][ 'media_viewer' ] = ( True, True, ( 640, 480 ), ( 70, 70 ), ( -1, -1 ), 'topleft', True, True )
@ -1753,7 +1754,7 @@ class ServiceRestricted( ServiceRemote ):
url = 'http://' + host + ':' + str( port ) + path_and_query
( response, size_of_response, response_headers, cookies ) = HydrusGlobals.client_controller.DoHTTP( method, url, request_headers, body, report_hooks = report_hooks, temp_path = temp_path, return_everything = True )
( response, size_of_response, response_headers, cookies ) = HydrusGlobals.client_controller.DoHTTP( method, url, request_headers, body, report_hooks = report_hooks, temp_path = temp_path, hydrus_network = True )
ClientNetworking.CheckHydrusVersion( self._service_key, self._service_type, response_headers )

70
include/ClientGUI.py
View File

@ -20,6 +20,7 @@ import ClientDownloading
import ClientMedia
import ClientSearch
import ClientThreading
import cv2
import gc
import HydrusData
import HydrusExceptions
@ -32,10 +33,13 @@ import HydrusNetworking
import HydrusSerialisable
import HydrusTagArchive
import HydrusThreading
import HydrusVideoHandling
import itertools
import os
import PIL
import random
import sqlite3
import ssl
import subprocess
import sys
import threading
@ -91,6 +95,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
self._message_manager = ClientGUICommon.PopupMessageManager( self )
self.Bind( wx.EVT_MIDDLE_DOWN, self.EventFrameMiddleClick )
self.Bind( wx.EVT_CLOSE, self.EventClose )
self.Bind( wx.EVT_MENU, self.EventMenu )
self.Bind( wx.EVT_SET_FOCUS, self.EventFocus )
@ -182,7 +187,37 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
aboutinfo.SetIcon( wx.Icon( os.path.join( HC.STATIC_DIR, 'hydrus.ico' ), wx.BITMAP_TYPE_ICO ) )
aboutinfo.SetName( 'hydrus client' )
aboutinfo.SetVersion( str( HC.SOFTWARE_VERSION ) + ', using network version ' + str( HC.NETWORK_VERSION ) )
aboutinfo.SetDescription( CC.CLIENT_DESCRIPTION )
library_versions = []
library_versions.append( ( 'FFMPEG', HydrusVideoHandling.GetFFMPEGVersion() ) )
library_versions.append( ( 'OpenCV', cv2.__version__ ) )
library_versions.append( ( 'openssl', ssl.OPENSSL_VERSION ) )
library_versions.append( ( 'PIL', PIL.VERSION ) )
if hasattr( PIL, 'PILLOW_VERSION' ):
library_versions.append( ( 'Pillow', PIL.PILLOW_VERSION ) )
# 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.1500 64 bit (AMD64)]
v = sys.version
if ' ' in v:
v = v.split( ' ' )[0]
library_versions.append( ( 'python', v ) )
library_versions.append( ( 'sqlite', sqlite3.sqlite_version ) )
library_versions.append( ( 'wx', wx.version() ) )
description = 'This client is the media management application of the hydrus software suite.'
description += os.linesep * 2 + os.linesep.join( ( lib + ': ' + version for ( lib, version ) in library_versions ) )
aboutinfo.SetDescription( description )
with open( os.path.join( HC.BASE_DIR, 'license.txt' ), 'rb' ) as f: license = f.read()
@ -553,6 +588,14 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _ChooseNewPage( self ):
with ClientGUIDialogs.DialogPageChooser( self ) as dlg:
dlg.ShowModal()
def _ClearOrphans( self ):
text = 'This will iterate through every file in your database\'s file storage, removing any it does not expect to be there. It may take some time.'
@ -1457,7 +1500,17 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
try:
with ClientGUIDialogsManage.DialogManageSubscriptions( self ) as dlg: dlg.ShowModal()
title = 'manage subscriptions'
frame_key = 'manage_subscriptions_dialog'
with ClientGUITopLevelWindows.DialogManage( self, title, frame_key ) as dlg:
panel = ClientGUIScrolledPanelsManagement.ManageSubscriptionsPanel( dlg )
dlg.SetPanel( panel )
dlg.ShowModal()
finally:
@ -2390,6 +2443,11 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
def EventFrameMiddleClick( self, event ):
self._ChooseNewPage()
def EventMenu( self, event ):
action = ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetAction( event.GetId() )
@ -2531,7 +2589,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
elif command == 'new_import_urls': self._NewPageImportURLs()
elif command == 'new_page':
with ClientGUIDialogs.DialogPageChooser( self ) as dlg: dlg.ShowModal()
self._ChooseNewPage()
elif command == 'new_page_query': self._NewPageQuery( data )
elif command == 'news': self._News( data )
@ -2598,7 +2656,11 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
( tab_index, flags ) = self._notebook.HitTest( ( event.GetX(), event.GetY() ) )
if tab_index != -1:
if tab_index == wx.NOT_FOUND:
self._ChooseNewPage()
else:
self._ClosePage( tab_index )

20
include/ClientGUICanvas.py
View File

@ -14,6 +14,7 @@ import ClientRatings
import ClientRendering
import collections
import gc
import HydrusExceptions
import HydrusImageHandling
import HydrusPaths
import HydrusTags
@ -2376,9 +2377,16 @@ class CanvasMediaList( ClientMedia.ListeningMediaList, CanvasWithDetails ):
def ProcessContentUpdates( self, service_keys_to_content_updates ):
next_media = self._GetNext( self._current_media )
if next_media == self._current_media: next_media = None
if self.HasMedia( self._current_media ):
next_media = self._GetNext( self._current_media )
if next_media == self._current_media: next_media = None
else:
next_media = None
ClientMedia.ListeningMediaList.ProcessContentUpdates( self, service_keys_to_content_updates )
@ -2392,10 +2400,14 @@ class CanvasMediaList( ClientMedia.ListeningMediaList, CanvasWithDetails ):
self._SetDirty()
else:
elif self.HasMedia( next_media ):
self.SetMedia( next_media )
else:
self.SetMedia( self._GetFirst() )
def TIMEREventCursorHide( self, event ):

6
include/ClientGUICollapsible.py
View File

@ -140,6 +140,10 @@ class CollapsiblePanel( wx.Panel ):
parent.Layout()
event = CC.SizeChangedEvent( -1 )
wx.CallAfter( self.ProcessEvent, event )
def IsExpanded( self ):
@ -154,4 +158,4 @@ class CollapsiblePanel( wx.Panel ):
self._panel.Hide()

549
include/ClientGUIDialogsManage.py
View File

@ -4857,7 +4857,9 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
return
except:
except Exception as e:
HydrusData.ShowException( e )
wx.MessageBox( 'Could not connect!' )
@ -5066,549 +5068,6 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
class DialogManageSubscriptions( ClientGUIDialogs.Dialog ):
def __init__( self, parent ):
ClientGUIDialogs.Dialog.__init__( self, parent, 'manage subscriptions' )
self._original_subscription_names = HydrusGlobals.client_controller.Read( 'serialisable_names', HydrusSerialisable.SERIALISABLE_TYPE_SUBSCRIPTION )
self._names_to_delete = set()
#
self._listbook = ClientGUICommon.ListBook( self )
self._add = wx.Button( self, label = 'add' )
self._add.Bind( wx.EVT_BUTTON, self.EventAdd )
self._add.SetForegroundColour( ( 0, 128, 0 ) )
self._remove = wx.Button( self, label = 'remove' )
self._remove.Bind( wx.EVT_BUTTON, self.EventRemove )
self._remove.SetForegroundColour( ( 128, 0, 0 ) )
self._export = wx.Button( self, label = 'export' )
self._export.Bind( wx.EVT_BUTTON, self.EventExport )
self._ok = wx.Button( self, id = wx.ID_OK, label = 'ok' )
self._ok.Bind( wx.EVT_BUTTON, self.EventOK )
self._ok.SetForegroundColour( ( 0, 128, 0 ) )
self._cancel = wx.Button( self, id = wx.ID_CANCEL, label = 'cancel' )
self._cancel.SetForegroundColour( ( 128, 0, 0 ) )
#
for name in self._original_subscription_names:
self._listbook.AddPageArgs( name, name, self._Panel, ( self._listbook, name ), {} )
#
text_hbox = wx.BoxSizer( wx.HORIZONTAL )
text_hbox.AddF( wx.StaticText( self, label = 'For more information about subscriptions, please check' ), CC.FLAGS_VCENTER )
text_hbox.AddF( wx.HyperlinkCtrl( self, id = -1, label = 'here', url = 'file://' + HC.HELP_DIR + '/getting_started_subscriptions.html' ), CC.FLAGS_VCENTER )
add_remove_hbox = wx.BoxSizer( wx.HORIZONTAL )
add_remove_hbox.AddF( self._add, CC.FLAGS_VCENTER )
add_remove_hbox.AddF( self._remove, CC.FLAGS_VCENTER )
add_remove_hbox.AddF( self._export, CC.FLAGS_VCENTER )
ok_hbox = wx.BoxSizer( wx.HORIZONTAL )
ok_hbox.AddF( self._ok, CC.FLAGS_VCENTER )
ok_hbox.AddF( self._cancel, CC.FLAGS_VCENTER )
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.AddF( text_hbox, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._listbook, CC.FLAGS_EXPAND_BOTH_WAYS )
vbox.AddF( add_remove_hbox, CC.FLAGS_SMALL_INDENT )
vbox.AddF( ok_hbox, CC.FLAGS_BUTTON_SIZER )
self.SetSizer( vbox )
#
( x, y ) = self.GetEffectiveMinSize()
self.SetInitialSize( ( 680, max( 720, y ) ) )
self.SetDropTarget( ClientDragDrop.FileDropTarget( self.Import ) )
wx.CallAfter( self._ok.SetFocus )
def EventAdd( self, event ):
with ClientGUIDialogs.DialogTextEntry( self, 'Enter a name for the subscription.' ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
try:
name = dlg.GetValue()
if self._listbook.KeyExists( name ):
raise HydrusExceptions.NameException( 'That name is already in use!' )
if name == '': raise HydrusExceptions.NameException( 'Please enter a nickname for the subscription.' )
page = self._Panel( self._listbook, name, is_new_subscription = True )
self._listbook.AddPage( name, name, page, select = True )
except HydrusExceptions.NameException as e:
wx.MessageBox( str( e ) )
self.EventAdd( event )
def EventExport( self, event ):
panel = self._listbook.GetCurrentPage()
if panel is not None:
subscription = panel.GetSubscription()
name = subscription.GetName()
dump = subscription.DumpToString()
try:
with wx.FileDialog( self, 'select where to export subscription', defaultFile = name + '.json', style = wx.FD_SAVE ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
path = HydrusData.ToUnicode( dlg.GetPath() )
with open( path, 'wb' ) as f: f.write( dump )
except:
with wx.FileDialog( self, 'select where to export subscription', defaultFile = 'subscription.json', style = wx.FD_SAVE ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
path = HydrusData.ToUnicode( dlg.GetPath() )
with open( path, 'wb' ) as f: f.write( dump )
def EventOK( self, event ):
all_pages = self._listbook.GetActivePages()
try:
for name in self._names_to_delete:
HydrusGlobals.client_controller.Write( 'delete_serialisable_named', HydrusSerialisable.SERIALISABLE_TYPE_SUBSCRIPTION, name )
for page in all_pages:
subscription = page.GetSubscription()
HydrusGlobals.client_controller.Write( 'serialisable', subscription )
HydrusGlobals.client_controller.pub( 'notify_new_subscriptions' )
finally: self.EndModal( wx.ID_OK )
def EventRemove( self, event ):
name = self._listbook.GetCurrentKey()
self._names_to_delete.add( name )
self._listbook.DeleteCurrentPage()
def Import( self, paths ):
for path in paths:
try:
with open( path, 'rb' ) as f: data = f.read()
subscription = HydrusSerialisable.CreateFromString( data )
name = subscription.GetName()
if self._listbook.KeyExists( name ):
message = 'A subscription with that name already exists. Overwrite it?'
with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
if dlg.ShowModal() == wx.ID_YES:
self._listbook.Select( name )
page = self._listbook.GetPage( name )
page.Update( subscription )
else:
page = self._Panel( self._listbook, name, is_new_subscription = True )
page.Update( subscription )
self._listbook.AddPage( name, name, page, select = True )
except:
wx.MessageBox( traceback.format_exc() )
class _Panel( wx.ScrolledWindow ):
def __init__( self, parent, name, is_new_subscription = False ):
wx.ScrolledWindow.__init__( self, parent )
self._is_new_subscription = is_new_subscription
if self._is_new_subscription:
self._original_subscription = ClientImporting.Subscription( name )
else:
self._original_subscription = HydrusGlobals.client_controller.Read( 'serialisable_named', HydrusSerialisable.SERIALISABLE_TYPE_SUBSCRIPTION, name )
#
self._query_panel = ClientGUICommon.StaticBox( self, 'site and query' )
self._site_type = ClientGUICommon.BetterChoice( self._query_panel )
site_types = []
site_types.append( HC.SITE_TYPE_BOORU )
site_types.append( HC.SITE_TYPE_DEVIANT_ART )
site_types.append( HC.SITE_TYPE_HENTAI_FOUNDRY_ARTIST )
site_types.append( HC.SITE_TYPE_HENTAI_FOUNDRY_TAGS )
site_types.append( HC.SITE_TYPE_NEWGROUNDS )
site_types.append( HC.SITE_TYPE_PIXIV_ARTIST_ID )
site_types.append( HC.SITE_TYPE_PIXIV_TAG )
site_types.append( HC.SITE_TYPE_TUMBLR )
for site_type in site_types:
self._site_type.Append( HC.site_type_string_lookup[ site_type ], site_type )
self._site_type.Bind( wx.EVT_CHOICE, self.EventSiteChanged )
self._query = wx.TextCtrl( self._query_panel )
self._booru_selector = wx.ListBox( self._query_panel )
self._booru_selector.Bind( wx.EVT_LISTBOX, self.EventBooruSelected )
self._period = ClientGUICommon.TimeDeltaButton( self._query_panel, min = 3600 * 4, days = True, hours = True )
self._info_panel = ClientGUICommon.StaticBox( self, 'info' )
self._get_tags_if_redundant = wx.CheckBox( self._info_panel, label = 'get tags even if new file is already in db' )
self._initial_file_limit = ClientGUICommon.NoneableSpinCtrl( self._info_panel, 'initial file limit', none_phrase = 'no limit', min = 1, max = 1000000 )
self._initial_file_limit.SetToolTipString( 'If set, the first sync will add no more than this many files. Otherwise, it will get everything the gallery has.' )
self._periodic_file_limit = ClientGUICommon.NoneableSpinCtrl( self._info_panel, 'periodic file limit', none_phrase = 'no limit', min = 1, max = 1000000 )
self._periodic_file_limit.SetToolTipString( 'If set, normal syncs will add no more than this many files. Otherwise, they will get everything up until they find a file they have seen before.' )
self._paused = wx.CheckBox( self._info_panel, label = 'paused' )
self._seed_cache_button = wx.BitmapButton( self._info_panel, bitmap = CC.GlobalBMPs.seed_cache )
self._seed_cache_button.Bind( wx.EVT_BUTTON, self.EventSeedCache )
self._seed_cache_button.SetToolTipString( 'open detailed url cache status' )
self._reset_cache_button = wx.Button( self._info_panel, label = ' reset url cache on dialog ok ' )
self._reset_cache_button.Bind( wx.EVT_BUTTON, self.EventResetCache )
self._check_now_button = wx.Button( self._info_panel, label = ' force sync on dialog ok ' )
self._check_now_button.Bind( wx.EVT_BUTTON, self.EventCheckNow )
self._import_tag_options = ClientGUICollapsible.CollapsibleOptionsTags( self )
self._import_file_options = ClientGUICollapsible.CollapsibleOptionsImportFiles( self )
#
self._SetControls()
#
hbox = wx.BoxSizer( wx.HORIZONTAL )
hbox.AddF( wx.StaticText( self._query_panel, label = 'Check subscription every ' ), CC.FLAGS_VCENTER )
hbox.AddF( self._period, CC.FLAGS_VCENTER )
self._query_panel.AddF( self._site_type, CC.FLAGS_EXPAND_PERPENDICULAR )
self._query_panel.AddF( self._query, CC.FLAGS_EXPAND_PERPENDICULAR )
self._query_panel.AddF( self._booru_selector, CC.FLAGS_EXPAND_PERPENDICULAR )
self._query_panel.AddF( hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
#
self._info_panel.AddF( self._get_tags_if_redundant, CC.FLAGS_LONE_BUTTON )
self._info_panel.AddF( self._initial_file_limit, CC.FLAGS_LONE_BUTTON )
self._info_panel.AddF( self._periodic_file_limit, CC.FLAGS_LONE_BUTTON )
self._info_panel.AddF( self._paused, CC.FLAGS_LONE_BUTTON )
last_checked_text = self._original_subscription.GetLastCheckedText()
self._info_panel.AddF( wx.StaticText( self._info_panel, label = last_checked_text ), CC.FLAGS_EXPAND_PERPENDICULAR )
seed_cache = self._original_subscription.GetSeedCache()
seed_cache_text = HydrusData.ConvertIntToPrettyString( seed_cache.GetSeedCount() ) + ' urls in cache'
num_failed = seed_cache.GetSeedCount( CC.STATUS_FAILED )
if num_failed > 0:
seed_cache_text += ', ' + HydrusData.ConvertIntToPrettyString( num_failed ) + ' failed'
self._info_panel.AddF( wx.StaticText( self._info_panel, label = seed_cache_text ), CC.FLAGS_EXPAND_PERPENDICULAR )
self._info_panel.AddF( self._seed_cache_button, CC.FLAGS_LONE_BUTTON )
self._info_panel.AddF( self._reset_cache_button, CC.FLAGS_LONE_BUTTON )
self._info_panel.AddF( self._check_now_button, CC.FLAGS_LONE_BUTTON )
#
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.AddF( self._query_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._info_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._import_tag_options, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._import_file_options, CC.FLAGS_EXPAND_PERPENDICULAR )
self.SetSizer( vbox )
self.SetScrollRate( 0, 20 )
( x, y ) = self.GetEffectiveMinSize()
self.SetInitialSize( ( x, y ) )
def _ConfigureImportTagOptions( self ):
gallery_identifier = self._GetGalleryIdentifier()
( namespaces, search_value ) = ClientDefaults.GetDefaultNamespacesAndSearchValue( gallery_identifier )
new_options = HydrusGlobals.client_controller.GetNewOptions()
import_tag_options = new_options.GetDefaultImportTagOptions( gallery_identifier )
if not self._is_new_subscription:
if gallery_identifier == self._original_subscription.GetGalleryIdentifier():
search_value = self._original_subscription.GetQuery()
import_tag_options = self._original_subscription.GetImportTagOptions()
self._query.SetValue( search_value )
self._import_tag_options.SetNamespaces( namespaces )
self._import_tag_options.SetOptions( import_tag_options )
def _GetGalleryIdentifier( self ):
site_type = self._site_type.GetChoice()
if site_type == HC.SITE_TYPE_BOORU:
booru_name = self._booru_selector.GetStringSelection()
gallery_identifier = ClientDownloading.GalleryIdentifier( site_type, additional_info = booru_name )
else:
gallery_identifier = ClientDownloading.GalleryIdentifier( site_type )
return gallery_identifier
def _PresentForSiteType( self ):
site_type = self._site_type.GetChoice()
if site_type == HC.SITE_TYPE_BOORU:
if self._booru_selector.GetCount() == 0:
boorus = HydrusGlobals.client_controller.Read( 'remote_boorus' )
for ( name, booru ) in boorus.items(): self._booru_selector.Append( name, booru )
self._booru_selector.Select( 0 )
self._booru_selector.Show()
else: self._booru_selector.Hide()
wx.CallAfter( self._ConfigureImportTagOptions )
self.Layout()
def _SetControls( self ):
( gallery_identifier, gallery_stream_identifiers, query, period, get_tags_if_redundant, initial_file_limit, periodic_file_limit, paused, import_file_options, import_tag_options ) = self._original_subscription.ToTuple()
site_type = gallery_identifier.GetSiteType()
self._site_type.SelectClientData( site_type )
self._PresentForSiteType()
if site_type == HC.SITE_TYPE_BOORU:
booru_name = gallery_identifier.GetAdditionalInfo()
index = self._booru_selector.FindString( booru_name )
if index != wx.NOT_FOUND:
self._booru_selector.Select( index )
# set gallery_stream_identifiers selection here--some kind of list of checkboxes or whatever
self._query.SetValue( query )
self._period.SetValue( period )
self._get_tags_if_redundant.SetValue( get_tags_if_redundant )
self._initial_file_limit.SetValue( initial_file_limit )
self._periodic_file_limit.SetValue( periodic_file_limit )
self._paused.SetValue( paused )
self._import_file_options.SetOptions( import_file_options )
self._import_tag_options.SetOptions( import_tag_options )
def EventBooruSelected( self, event ):
self._ConfigureImportTagOptions()
def EventCheckNow( self, event ):
self._original_subscription.CheckNow()
self._check_now_button.SetLabelText( 'will check on dialog ok' )
self._check_now_button.Disable()
def EventResetCache( self, event ):
message = '''Resetting this subscription's cache will delete ''' + HydrusData.ConvertIntToPrettyString( self._original_subscription.GetSeedCache().GetSeedCount() ) + ''' remembered urls, meaning when the subscription next runs, it will try to download those all over again. This may be expensive in time and data. Only do it if you are willing to wait. Do you want to do it?'''
with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
if dlg.ShowModal() == wx.ID_YES:
self._original_subscription.Reset()
self._reset_cache_button.SetLabelText( 'cache will be reset on dialog ok' )
self._reset_cache_button.Disable()
def EventSeedCache( self, event ):
seed_cache = self._original_subscription.GetSeedCache()
dupe_seed_cache = seed_cache.Duplicate()
with ClientGUITopLevelWindows.DialogEdit( self, 'file import status' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditSeedCachePanel( dlg, HydrusGlobals.client_controller, dupe_seed_cache )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
self._original_subscription.SetSeedCache( dupe_seed_cache )
def EventSiteChanged( self, event ): self._PresentForSiteType()
def GetSubscription( self ):
gallery_identifier = self._GetGalleryIdentifier()
# in future, this can be harvested from some checkboxes or whatever for stream selection
gallery_stream_identifiers = ClientDownloading.GetGalleryStreamIdentifiers( gallery_identifier )
query = self._query.GetValue()
period = self._period.GetValue()
get_tags_if_redundant = self._get_tags_if_redundant.GetValue()
initial_file_limit = self._initial_file_limit.GetValue()
periodic_file_limit = self._periodic_file_limit.GetValue()
paused = self._paused.GetValue()
import_file_options = self._import_file_options.GetOptions()
import_tag_options = self._import_tag_options.GetOptions()
self._original_subscription.SetTuple( gallery_identifier, gallery_stream_identifiers, query, period, get_tags_if_redundant, initial_file_limit, periodic_file_limit, paused, import_file_options, import_tag_options )
return self._original_subscription
def Update( self, subscription ):
self._original_subscription = subscription
self._SetControls()
class DialogManageTagCensorship( ClientGUIDialogs.Dialog ):
def __init__( self, parent, initial_value = None ):
@ -7201,4 +6660,4 @@ class DialogManageUPnP( ClientGUIDialogs.Dialog ):
if do_refresh: self._RefreshMappings()

2
include/ClientGUIManagement.py
View File

@ -1480,7 +1480,7 @@ class ManagementPanelGalleryImport( ManagementPanel ):
self._get_tags_if_redundant.Bind( wx.EVT_CHECKBOX, self.EventGetTagsIfRedundant )
self._get_tags_if_redundant.SetToolTipString( 'only fetch tags from the gallery if the file is new' )
self._file_limit = ClientGUICommon.NoneableSpinCtrl( self._gallery_downloader_panel, 'file limit', min = 1 )
self._file_limit = ClientGUICommon.NoneableSpinCtrl( self._gallery_downloader_panel, 'stop searching once this many files are found', min = 1, none_phrase = 'no limit' )
self._file_limit.Bind( wx.EVT_SPINCTRL, self.EventFileLimit )
self._file_limit.SetToolTipString( 'per query, stop searching the gallery once this many files has been reached' )

3
include/ClientGUIMedia.py
View File

@ -14,7 +14,6 @@ import collections
import HydrusExceptions
import HydrusPaths
import HydrusSerialisable
import HydrusTagArchive
import HydrusTags
import HydrusThreading
import itertools
@ -3415,4 +3414,4 @@ class ThumbnailMediaSingleton( Thumbnail, ClientMedia.MediaSingleton ):
ClientMedia.MediaSingleton.__init__( self, media_result )
Thumbnail.__init__( self, file_service_key )

108
include/ClientGUIParsing.py
View File

@ -1538,7 +1538,7 @@ class ManageParsingScriptsPanel( ClientGUIScrolledPanels.ManagePanel ):
menu_items.append( ( 'from clipboard', 'Load a script from text in your clipboard.', self.ImportFromClipboard ) )
menu_items.append( ( 'from png', 'Load a script from an encoded png.', self.ImportFromPng ) )
self._paste_button = ClientGUICommon.MenuButton( self, 'import', menu_items )
self._import_button = ClientGUICommon.MenuButton( self, 'import', menu_items )
self._duplicate_button = ClientGUICommon.BetterButton( self, 'duplicate', self.Duplicate )
@ -1565,7 +1565,7 @@ class ManageParsingScriptsPanel( ClientGUIScrolledPanels.ManagePanel ):
button_hbox.AddF( self._add_button, CC.FLAGS_VCENTER )
button_hbox.AddF( self._export_button, CC.FLAGS_VCENTER )
button_hbox.AddF( self._paste_button, CC.FLAGS_VCENTER )
button_hbox.AddF( self._import_button, CC.FLAGS_VCENTER )
button_hbox.AddF( self._duplicate_button, CC.FLAGS_VCENTER )
button_hbox.AddF( self._edit_button, CC.FLAGS_VCENTER )
button_hbox.AddF( self._delete_button, CC.FLAGS_VCENTER )
@ -1583,6 +1583,59 @@ class ManageParsingScriptsPanel( ClientGUIScrolledPanels.ManagePanel ):
return ( ( name, query_type, script_type, produces ), ( script, query_type, script_type, produces ) )
def _GetExportObject( self ):
to_export = HydrusSerialisable.SerialisableList()
for i in self._scripts.GetAllSelected():
single_object = self._scripts.GetClientData( i )[0]
to_export.append( single_object )
if len( to_export ) == 0:
return None
elif len( to_export ) == 1:
return to_export[0]
else:
return to_export
def _ImportObject( self, obj ):
if isinstance( obj, HydrusSerialisable.SerialisableList ):
for sub_obj in obj:
self._ImportObject( sub_obj )
else:
if isinstance( obj, ClientParsing.ParseRootFileLookup ):
script = obj
self._SetNonDupeName( script )
( display_tuple, data_tuple ) = self._ConvertScriptToTuples( script )
self._scripts.Append( display_tuple, data_tuple )
else:
wx.MessageBox( 'That was not a script--it was a: ' + type( obj ).__name__ )
def _SetNonDupeName( self, script ):
name = script.GetName()
@ -1748,25 +1801,25 @@ class ManageParsingScriptsPanel( ClientGUIScrolledPanels.ManagePanel ):
def ExportToClipboard( self ):
for i in self._scripts.GetAllSelected():
export_object = self._GetExportObject()
if export_object is not None:
( script, query_type, script_type, produces ) = self._scripts.GetClientData( i )
json = export_object.DumpToString()
script_json = script.DumpToString()
HydrusGlobals.client_controller.pub( 'clipboard', 'text', script_json )
HydrusGlobals.client_controller.pub( 'clipboard', 'text', json )
def ExportToPng( self ):
for i in self._scripts.GetAllSelected():
export_object = self._GetExportObject()
if export_object is not None:
( script, query_type, script_type, produces ) = self._scripts.GetClientData( i )
with ClientGUITopLevelWindows.DialogNullipotent( self, 'export script to png' ) as dlg:
with ClientGUITopLevelWindows.DialogNullipotent( self, 'export to png' ) as dlg:
panel = ClientGUISerialisable.PngExportPanel( dlg, script )
panel = ClientGUISerialisable.PngExportPanel( dlg, export_object )
dlg.SetPanel( panel )
@ -1791,20 +1844,7 @@ class ManageParsingScriptsPanel( ClientGUIScrolledPanels.ManagePanel ):
obj = HydrusSerialisable.CreateFromString( raw_text )
if isinstance( obj, ClientParsing.ParseRootFileLookup ):
script = obj
self._SetNonDupeName( script )
( display_tuple, data_tuple ) = self._ConvertScriptToTuples( script )
self._scripts.Append( display_tuple, data_tuple )
else:
wx.MessageBox( 'That was not a script--it was a: ' + type( obj ).__name__ )
self._ImportObject( obj )
except Exception as e:
@ -1840,20 +1880,7 @@ class ManageParsingScriptsPanel( ClientGUIScrolledPanels.ManagePanel ):
obj = HydrusSerialisable.CreateFromNetworkString( payload )
if isinstance( obj, ClientParsing.ParseRootFileLookup ):
script = obj
self._SetNonDupeName( script )
( display_tuple, data_tuple ) = self._ConvertScriptToTuples( script )
self._scripts.Append( display_tuple, data_tuple )
else:
wx.MessageBox( 'That was not a script--it was a: ' + type( obj ).__name__ )
self._ImportObject( obj )
except:
@ -1863,7 +1890,6 @@ class ManageParsingScriptsPanel( ClientGUIScrolledPanels.ManagePanel ):
class ScriptManagementControl( wx.Panel ):
def __init__( self, parent ):

472
include/ClientGUIScrolledPanelsEdit.py
View File

@ -1,7 +1,15 @@
import ClientConstants as CC
import ClientDefaults
import ClientDownloading
import ClientImporting
import ClientGUICollapsible
import ClientGUICommon
import ClientGUIDialogs
import ClientGUIScrolledPanels
import ClientGUITopLevelWindows
import HydrusConstants as HC
import HydrusData
import HydrusGlobals
import wx
class EditFrameLocationPanel( ClientGUIScrolledPanels.EditPanel ):
@ -324,4 +332,466 @@ class EditSeedCachePanel( ClientGUIScrolledPanels.EditPanel ):
self._UpdateText()
class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
def __init__( self, parent, subscription ):
subscription = subscription.Duplicate()
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
self._original_subscription = subscription
#
self._name = wx.TextCtrl( self )
#
self._info_panel = ClientGUICommon.StaticBox( self, 'info' )
self._last_checked_st = wx.StaticText( self._info_panel )
self._next_check_st = wx.StaticText( self._info_panel )
self._seed_info_st = wx.StaticText( self._info_panel )
#
self._query_panel = ClientGUICommon.StaticBox( self, 'site and query' )
self._site_type = ClientGUICommon.BetterChoice( self._query_panel )
site_types = []
site_types.append( HC.SITE_TYPE_BOORU )
site_types.append( HC.SITE_TYPE_DEVIANT_ART )
site_types.append( HC.SITE_TYPE_HENTAI_FOUNDRY_ARTIST )
site_types.append( HC.SITE_TYPE_HENTAI_FOUNDRY_TAGS )
site_types.append( HC.SITE_TYPE_NEWGROUNDS )
site_types.append( HC.SITE_TYPE_PIXIV_ARTIST_ID )
site_types.append( HC.SITE_TYPE_PIXIV_TAG )
site_types.append( HC.SITE_TYPE_TUMBLR )
for site_type in site_types:
self._site_type.Append( HC.site_type_string_lookup[ site_type ], site_type )
self._site_type.Bind( wx.EVT_CHOICE, self.EventSiteChanged )
self._query = wx.TextCtrl( self._query_panel )
self._booru_selector = wx.ListBox( self._query_panel )
self._booru_selector.Bind( wx.EVT_LISTBOX, self.EventBooruSelected )
self._period = ClientGUICommon.TimeDeltaButton( self._query_panel, min = 3600 * 4, days = True, hours = True )
self._period.Bind( ClientGUICommon.EVT_TIME_DELTA, self.EventPeriodChanged )
#
self._options_panel = ClientGUICommon.StaticBox( self, 'options' )
self._get_tags_if_redundant = wx.CheckBox( self._options_panel )
self._initial_file_limit = ClientGUICommon.NoneableSpinCtrl( self._options_panel, '', none_phrase = 'get everything', min = 1, max = 1000000 )
self._initial_file_limit.SetToolTipString( 'If set, the first sync will add no more than this many files. Otherwise, it will get everything the gallery has.' )
self._periodic_file_limit = ClientGUICommon.NoneableSpinCtrl( self._options_panel, '', none_phrase = 'get everything', min = 1, max = 1000000 )
self._periodic_file_limit.SetToolTipString( 'If set, normal syncs will add no more than this many files. Otherwise, they will get everything up until they find a file they have seen before.' )
#
self._control_panel = ClientGUICommon.StaticBox( self, 'control' )
self._paused = wx.CheckBox( self._control_panel )
self._seed_cache_button = wx.BitmapButton( self._control_panel, bitmap = CC.GlobalBMPs.seed_cache )
self._seed_cache_button.Bind( wx.EVT_BUTTON, self.EventSeedCache )
self._seed_cache_button.SetToolTipString( 'open detailed url cache status' )
self._retry_failed = ClientGUICommon.BetterButton( self._control_panel, 'retry failed', self.RetryFailed )
self._check_now_button = ClientGUICommon.BetterButton( self._control_panel, 'force check on dialog ok', self.CheckNow )
self._reset_cache_button = ClientGUICommon.BetterButton( self._control_panel, 'reset url cache', self.ResetCache )
#
self._import_tag_options = ClientGUICollapsible.CollapsibleOptionsTags( self )
self._import_file_options = ClientGUICollapsible.CollapsibleOptionsImportFiles( self )
#
name = subscription.GetName()
self._name.SetValue( name )
( gallery_identifier, gallery_stream_identifiers, query, period, get_tags_if_redundant, initial_file_limit, periodic_file_limit, paused, import_file_options, import_tag_options, self._last_checked, self._last_error, self._check_now, self._seed_cache ) = subscription.ToTuple()
site_type = gallery_identifier.GetSiteType()
self._site_type.SelectClientData( site_type )
self._PresentForSiteType()
if site_type == HC.SITE_TYPE_BOORU:
booru_name = gallery_identifier.GetAdditionalInfo()
index = self._booru_selector.FindString( booru_name )
if index != wx.NOT_FOUND:
self._booru_selector.Select( index )
# set gallery_stream_identifiers selection here--some kind of list of checkboxes or whatever
self._query.SetValue( query )
self._period.SetValue( period )
self._get_tags_if_redundant.SetValue( get_tags_if_redundant )
self._initial_file_limit.SetValue( initial_file_limit )
self._periodic_file_limit.SetValue( periodic_file_limit )
self._paused.SetValue( paused )
self._import_file_options.SetOptions( import_file_options )
self._import_tag_options.SetOptions( import_tag_options )
if self._last_checked == 0:
self._reset_cache_button.Disable()
if self._check_now:
self._check_now_button.Disable()
self._UpdateCommandButtons()
self._UpdateLastNextCheck()
self._UpdateSeedInfo()
#
self._info_panel.AddF( self._last_checked_st, CC.FLAGS_EXPAND_PERPENDICULAR )
self._info_panel.AddF( self._next_check_st, CC.FLAGS_EXPAND_PERPENDICULAR )
self._info_panel.AddF( self._seed_info_st, CC.FLAGS_EXPAND_PERPENDICULAR )
#
rows = []
rows.append( ( 'search text: ', self._query ) )
rows.append( ( 'check for new files every: ', self._period ) )
gridbox = ClientGUICommon.WrapInGrid( self._query_panel, rows )
self._query_panel.AddF( self._site_type, CC.FLAGS_EXPAND_PERPENDICULAR )
self._query_panel.AddF( self._booru_selector, CC.FLAGS_EXPAND_PERPENDICULAR )
self._query_panel.AddF( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
#
rows = []
rows.append( ( 'get tags even if new file is already in db: ', self._get_tags_if_redundant ) )
rows.append( ( 'on first check, get at most this many files: ', self._initial_file_limit ) )
rows.append( ( 'on normal checks, get at most this many newer files: ', self._periodic_file_limit ) )
gridbox = ClientGUICommon.WrapInGrid( self._options_panel, rows )
self._options_panel.AddF( gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
#
self._control_panel.AddF( self._seed_cache_button, CC.FLAGS_LONE_BUTTON )
rows = []
rows.append( ( 'currently paused: ', self._paused ) )
gridbox = ClientGUICommon.WrapInGrid( self._control_panel, rows )
self._control_panel.AddF( gridbox, CC.FLAGS_LONE_BUTTON )
self._control_panel.AddF( self._retry_failed, CC.FLAGS_LONE_BUTTON )
self._control_panel.AddF( self._check_now_button, CC.FLAGS_LONE_BUTTON )
self._control_panel.AddF( self._reset_cache_button, CC.FLAGS_LONE_BUTTON )
#
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.AddF( ClientGUICommon.WrapInText( self._name, self, 'name: ' ), CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
vbox.AddF( self._info_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._query_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._options_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._control_panel, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._import_tag_options, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._import_file_options, CC.FLAGS_EXPAND_PERPENDICULAR )
self.SetSizer( vbox )
def _ConfigureImportTagOptions( self ):
gallery_identifier = self._GetGalleryIdentifier()
( namespaces, search_value ) = ClientDefaults.GetDefaultNamespacesAndSearchValue( gallery_identifier )
new_options = HydrusGlobals.client_controller.GetNewOptions()
import_tag_options = new_options.GetDefaultImportTagOptions( gallery_identifier )
if gallery_identifier == self._original_subscription.GetGalleryIdentifier():
search_value = self._original_subscription.GetQuery()
import_tag_options = self._original_subscription.GetImportTagOptions()
self._query.SetValue( search_value )
self._import_tag_options.SetNamespaces( namespaces )
self._import_tag_options.SetOptions( import_tag_options )
def _GetGalleryIdentifier( self ):
site_type = self._site_type.GetChoice()
if site_type == HC.SITE_TYPE_BOORU:
booru_name = self._booru_selector.GetStringSelection()
gallery_identifier = ClientDownloading.GalleryIdentifier( site_type, additional_info = booru_name )
else:
gallery_identifier = ClientDownloading.GalleryIdentifier( site_type )
return gallery_identifier
def _UpdateCommandButtons( self ):
on_initial_sync = self._last_checked == 0
no_failures = self._seed_cache.GetSeedCount( CC.STATUS_FAILED ) == 0
can_check = not ( self._check_now or on_initial_sync )
if no_failures:
self._retry_failed.Disable()
else:
self._retry_failed.Enable()
if can_check:
self._check_now_button.Enable()
else:
self._check_now_button.Disable()
if on_initial_sync:
self._reset_cache_button.Disable()
else:
self._reset_cache_button.Enable()
def _UpdateLastNextCheck( self ):
if self._last_checked == 0:
last_checked_text = 'initial check has not yet occured'
else:
last_checked_text = 'last checked ' + HydrusData.ConvertTimestampToPrettySync( self._last_checked )
self._last_checked_st.SetLabelText( last_checked_text )
periodic_next_check_time = self._last_checked + self._period.GetValue()
error_next_check_time = self._last_error + HC.UPDATE_DURATION
if self._check_now:
next_check_text = 'next check as soon as manage subscriptions dialog is closed'
elif error_next_check_time > periodic_next_check_time:
next_check_text = 'due to an error, next check ' + HydrusData.ConvertTimestampToPrettyPending( error_next_check_time )
else:
next_check_text = 'next check ' + HydrusData.ConvertTimestampToPrettyPending( periodic_next_check_time )
self._next_check_st.SetLabelText( next_check_text )
def _UpdateSeedInfo( self ):
seed_cache_text = HydrusData.ConvertIntToPrettyString( self._seed_cache.GetSeedCount() ) + ' urls in cache'
num_failed = self._seed_cache.GetSeedCount( CC.STATUS_FAILED )
if num_failed > 0:
seed_cache_text += ', ' + HydrusData.ConvertIntToPrettyString( num_failed ) + ' failed'
self._seed_info_st.SetLabelText( seed_cache_text )
def _PresentForSiteType( self ):
site_type = self._site_type.GetChoice()
if site_type == HC.SITE_TYPE_BOORU:
if self._booru_selector.GetCount() == 0:
boorus = HydrusGlobals.client_controller.Read( 'remote_boorus' )
for ( name, booru ) in boorus.items(): self._booru_selector.Append( name, booru )
self._booru_selector.Select( 0 )
self._booru_selector.Show()
else:
self._booru_selector.Hide()
wx.CallAfter( self._ConfigureImportTagOptions )
event = CC.SizeChangedEvent( -1 )
wx.CallAfter( self.ProcessEvent, event )
def CheckNow( self, event ):
self._check_now = True
self._UpdateCommandButtons()
self._UpdateLastNextCheck()
def EventBooruSelected( self, event ):
self._ConfigureImportTagOptions()
def EventPeriodChanged( self, event ):
self._UpdateLastNextCheck()
def EventSeedCache( self, event ):
dupe_seed_cache = self._seed_cache.Duplicate()
with ClientGUITopLevelWindows.DialogEdit( self, 'file import status' ) as dlg:
panel = EditSeedCachePanel( dlg, HydrusGlobals.client_controller, dupe_seed_cache )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
self._seed_cache = panel.GetValue()
self._UpdateCommandButtons()
self._UpdateSeedInfo()
def EventSiteChanged( self, event ):
self._PresentForSiteType()
def GetValue( self ):
name = self._name.GetValue()
subscription = ClientImporting.Subscription( name )
gallery_identifier = self._GetGalleryIdentifier()
# in future, this can be harvested from some checkboxes or whatever for stream selection
gallery_stream_identifiers = ClientDownloading.GetGalleryStreamIdentifiers( gallery_identifier )
query = self._query.GetValue()
period = self._period.GetValue()
get_tags_if_redundant = self._get_tags_if_redundant.GetValue()
initial_file_limit = self._initial_file_limit.GetValue()
periodic_file_limit = self._periodic_file_limit.GetValue()
paused = self._paused.GetValue()
import_file_options = self._import_file_options.GetOptions()
import_tag_options = self._import_tag_options.GetOptions()
subscription.SetTuple( gallery_identifier, gallery_stream_identifiers, query, period, get_tags_if_redundant, initial_file_limit, periodic_file_limit, paused, import_file_options, import_tag_options, self._last_checked, self._last_error, self._check_now, self._seed_cache )
return subscription
def ResetCache( self, event ):
message = '''Resetting this subscription's cache will delete ''' + HydrusData.ConvertIntToPrettyString( self._original_subscription.GetSeedCache().GetSeedCount() ) + ''' remembered urls, meaning when the subscription next runs, it will try to download those all over again. This may be expensive in time and data. Only do it if you are willing to wait. Do you want to do it?'''
with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
if dlg.ShowModal() == wx.ID_YES:
self._last_checked = 0
self._last_error = 0
self._seed_cache = ClientImporting.SeedCache()
self._UpdateCommandButtons()
self._UpdateLastNextCheck()
self._UpdateSeedInfo()
def RetryFailed( self ):
failed_seeds = self._seed_cache.GetSeeds( CC.STATUS_FAILED )
for seed in failed_seeds:
self._seed_cache.UpdateSeedStatus( seed, CC.STATUS_UNKNOWN )
self._last_error = 0
self._UpdateCommandButtons()
self._UpdateLastNextCheck()
self._UpdateSeedInfo()

465
include/ClientGUIScrolledPanelsManagement.py
View File

@ -8,9 +8,12 @@ import ClientGUIDialogs
import ClientGUIPredicates
import ClientGUIScrolledPanels
import ClientGUIScrolledPanelsEdit
import ClientGUISerialisable
import ClientGUITagSuggestions
import ClientGUITopLevelWindows
import ClientImporting
import ClientMedia
import ClientSerialisable
import collections
import HydrusConstants as HC
import HydrusData
@ -519,7 +522,7 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
gallery_downloader = ClientGUICommon.StaticBox( self, 'gallery downloader' )
self._gallery_file_limit = ClientGUICommon.NoneableSpinCtrl( gallery_downloader, 'default file limit', none_phrase = 'no limit', min = 1, max = 1000000 )
self._gallery_file_limit = ClientGUICommon.NoneableSpinCtrl( gallery_downloader, 'by default, stop searching once this many files are found', none_phrase = 'no limit', min = 1, max = 1000000 )
#
@ -1126,8 +1129,6 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._gui_capitalisation.SetValue( HC.options[ 'gui_capitalisation' ] )
remember_tuple = self._new_options.GetFrameLocation( 'manage_tags_dialog' )
self._hide_preview.SetValue( HC.options[ 'hide_preview' ] )
self._show_thumbnail_title_banner.SetValue( self._new_options.GetBoolean( 'show_thumbnail_title_banner' ) )
@ -2344,6 +2345,462 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
wx.MessageBox( traceback.format_exc() )
class ManageSubscriptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
def __init__( self, parent ):
ClientGUIScrolledPanels.ManagePanel.__init__( self, parent )
subscriptions = HydrusGlobals.client_controller.Read( 'serialisable_named', HydrusSerialisable.SERIALISABLE_TYPE_SUBSCRIPTION )
#
columns = [ ( 'name', -1 ), ( 'site', 80 ), ( 'period', 80 ), ( 'last checked', 100 ), ( 'recent error?', 100 ), ( 'urls', 60 ), ( 'failures', 60 ), ( 'paused', 80 ), ( 'check now?', 100 ) ]
self._subscriptions = ClientGUICommon.SaneListCtrl( self, 300, columns, delete_key_callback = self.Delete, activation_callback = self.Edit, use_display_tuple_for_sort = True )
self._add = ClientGUICommon.BetterButton( self, 'add', self.Add )
menu_items = []
menu_items.append( ( 'to clipboard', 'Serialise the script and put it on your clipboard.', self.ExportToClipboard ) )
menu_items.append( ( 'to png', 'Serialise the script and encode it to an image file you can easily share with other hydrus users.', self.ExportToPng ) )
self._export = ClientGUICommon.MenuButton( self, 'export', menu_items )
menu_items = []
menu_items.append( ( 'from clipboard', 'Load a script from text in your clipboard.', self.ImportFromClipboard ) )
menu_items.append( ( 'from png', 'Load a script from an encoded png.', self.ImportFromPng ) )
self._import = ClientGUICommon.MenuButton( self, 'import', menu_items )
self._duplicate = ClientGUICommon.BetterButton( self, 'duplicate', self.Duplicate )
self._edit = ClientGUICommon.BetterButton( self, 'edit', self.Edit )
self._delete = ClientGUICommon.BetterButton( self, 'delete', self.Delete )
self._retry_failures = ClientGUICommon.BetterButton( self, 'retry failures', self.RetryFailures )
self._pause_resume = ClientGUICommon.BetterButton( self, 'pause/resume', self.PauseResume )
self._check_now = ClientGUICommon.BetterButton( self, 'check now', self.CheckNow )
self._reset = ClientGUICommon.BetterButton( self, 'reset', self.Reset )
#
for subscription in subscriptions:
( display_tuple, data_tuple ) = self._ConvertSubscriptionToTuples( subscription )
self._subscriptions.Append( display_tuple, data_tuple )
#
text_hbox = wx.BoxSizer( wx.HORIZONTAL )
text_hbox.AddF( wx.StaticText( self, label = 'For more information about subscriptions, please check' ), CC.FLAGS_VCENTER )
text_hbox.AddF( wx.HyperlinkCtrl( self, id = -1, label = 'here', url = 'file://' + HC.HELP_DIR + '/getting_started_subscriptions.html' ), CC.FLAGS_VCENTER )
action_box = wx.BoxSizer( wx.HORIZONTAL )
action_box.AddF( self._retry_failures, CC.FLAGS_VCENTER )
action_box.AddF( self._pause_resume, CC.FLAGS_VCENTER )
action_box.AddF( self._check_now, CC.FLAGS_VCENTER )
action_box.AddF( self._reset, CC.FLAGS_VCENTER )
button_box = wx.BoxSizer( wx.HORIZONTAL )
button_box.AddF( self._add, CC.FLAGS_VCENTER )
button_box.AddF( self._export, CC.FLAGS_VCENTER )
button_box.AddF( self._import, CC.FLAGS_VCENTER )
button_box.AddF( self._duplicate, CC.FLAGS_VCENTER )
button_box.AddF( self._edit, CC.FLAGS_VCENTER )
button_box.AddF( self._delete, CC.FLAGS_VCENTER )
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.AddF( text_hbox, CC.FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._subscriptions, CC.FLAGS_EXPAND_BOTH_WAYS )
vbox.AddF( action_box, CC.FLAGS_BUTTON_SIZER )
vbox.AddF( button_box, CC.FLAGS_BUTTON_SIZER )
self.SetSizer( vbox )
def _ConvertSubscriptionToTuples( self, subscription ):
( name, site, period, last_checked, recent_error, urls, failures, paused, check_now ) = subscription.ToPrettyStrings()
return ( ( name, site, period, last_checked, recent_error, urls, failures, paused, check_now ), ( subscription, site, period, last_checked, recent_error, urls, failures, paused, check_now ) )
def _GetExportObject( self ):
to_export = HydrusSerialisable.SerialisableList()
for subscription in self._GetSubscriptions( only_selected = True ):
to_export.append( subscription )
if len( to_export ) == 0:
return None
elif len( to_export ) == 1:
return to_export[0]
else:
return to_export
def _GetSubscriptions( self, only_selected = False ):
subscriptions = []
if only_selected:
for i in self._subscriptions.GetAllSelected():
subscription = self._subscriptions.GetClientData( i )[0]
subscriptions.append( subscription )
else:
for row in self._subscriptions.GetClientData():
subscription = row[0]
subscriptions.append( subscription )
return subscriptions
def _ImportObject( self, obj ):
if isinstance( obj, HydrusSerialisable.SerialisableList ):
for sub_obj in obj:
self._ImportObject( sub_obj )
else:
if isinstance( obj, ClientImporting.Subscription ):
subscription = obj
self._SetNonDupeName( subscription )
( display_tuple, data_tuple ) = self._ConvertSubscriptionToTuples( subscription )
self._subscriptions.Append( display_tuple, data_tuple )
else:
wx.MessageBox( 'That was not a script--it was a: ' + type( obj ).__name__ )
def _SetNonDupeName( self, subscription ):
name = subscription.GetName()
current_names = { s.GetName() for s in self._GetSubscriptions() }
if name in current_names:
i = 1
original_name = name
while name in current_names:
name = original_name + ' (' + str( i ) + ')'
i += 1
subscription.SetName( name )
def Add( self ):
empty_subscription = ClientImporting.Subscription( 'new subscription' )
with ClientGUITopLevelWindows.DialogEdit( self, 'edit subscription' ) as dlg_edit:
panel = ClientGUIScrolledPanelsEdit.EditSubscriptionPanel( dlg_edit, empty_subscription )
dlg_edit.SetPanel( panel )
if dlg_edit.ShowModal() == wx.ID_OK:
new_subscription = panel.GetValue()
self._SetNonDupeName( new_subscription )
( display_tuple, data_tuple ) = self._ConvertSubscriptionToTuples( new_subscription )
self._subscriptions.Append( display_tuple, data_tuple )
def CheckNow( self ):
for i in self._subscriptions.GetAllSelected():
subscription = self._subscriptions.GetClientData( i )[0]
subscription.CheckNow()
( display_tuple, data_tuple ) = self._ConvertSubscriptionToTuples( subscription )
self._subscriptions.UpdateRow( i, display_tuple, data_tuple )
def CommitChanges( self ):
existing_db_names = set( HydrusGlobals.client_controller.Read( 'serialisable_names', HydrusSerialisable.SERIALISABLE_TYPE_SUBSCRIPTION ) )
subscriptions = self._GetSubscriptions()
save_names = { subscription.GetName() for subscription in subscriptions }
deletee_names = existing_db_names.difference( save_names )
for subscription in subscriptions:
HydrusGlobals.client_controller.Write( 'serialisable', subscription )
for name in deletee_names:
HydrusGlobals.client_controller.Write( 'delete_serialisable_named', HydrusSerialisable.SERIALISABLE_TYPE_SUBSCRIPTION, name )
HydrusGlobals.client_controller.pub( 'notify_new_subscriptions' )
def Delete( self ):
self._subscriptions.RemoveAllSelected()
def Duplicate( self ):
subs_to_dupe = []
for subscription in self._GetSubscriptions( only_selected = True ):
subs_to_dupe.append( subscription )
for subscription in subs_to_dupe:
dupe_subscription = subscription.Duplicate()
self._SetNonDupeName( dupe_subscription )
( display_tuple, data_tuple ) = self._ConvertSubscriptionToTuples( dupe_subscription )
self._subscriptions.Append( display_tuple, data_tuple )
def Edit( self ):
for i in self._subscriptions.GetAllSelected():
subscription = self._subscriptions.GetClientData( i )[0]
with ClientGUITopLevelWindows.DialogEdit( self, 'edit subscription' ) as dlg:
original_name = subscription.GetName()
panel = ClientGUIScrolledPanelsEdit.EditSubscriptionPanel( dlg, subscription )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
edited_subscription = panel.GetValue()
name = edited_subscription.GetName()
if name != original_name:
self._SetNonDupeName( edited_subscription )
( display_tuple, data_tuple ) = self._ConvertSubscriptionToTuples( edited_subscription )
self._subscriptions.UpdateRow( i, display_tuple, data_tuple )
def ExportToClipboard( self ):
export_object = self._GetExportObject()
if export_object is not None:
json = export_object.DumpToString()
HydrusGlobals.client_controller.pub( 'clipboard', 'text', json )
def ExportToPng( self ):
export_object = self._GetExportObject()
if export_object is not None:
with ClientGUITopLevelWindows.DialogNullipotent( self, 'export to png' ) as dlg:
panel = ClientGUISerialisable.PngExportPanel( dlg, export_object )
dlg.SetPanel( panel )
dlg.ShowModal()
def ImportFromClipboard( self ):
if wx.TheClipboard.Open():
data = wx.TextDataObject()
wx.TheClipboard.GetData( data )
wx.TheClipboard.Close()
raw_text = data.GetText()
try:
obj = HydrusSerialisable.CreateFromString( raw_text )
self._ImportObject( obj )
except Exception as e:
wx.MessageBox( 'I could not understand what was in the clipboard' )
else:
wx.MessageBox( 'I could not get permission to access the clipboard.' )
def ImportFromPng( self ):
with wx.FileDialog( self, 'select the png with the encoded script', wildcard = 'PNG (*.png)|*.png' ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
path = HydrusData.ToUnicode( dlg.GetPath() )
try:
payload = ClientSerialisable.LoadFromPng( path )
except Exception as e:
wx.MessageBox( str( e ) )
return
try:
obj = HydrusSerialisable.CreateFromNetworkString( payload )
self._ImportObject( obj )
except:
wx.MessageBox( 'I could not understand what was encoded in the png!' )
def PauseResume( self ):
for i in self._subscriptions.GetAllSelected():
subscription = self._subscriptions.GetClientData( i )[0]
subscription.PauseResume()
( display_tuple, data_tuple ) = self._ConvertSubscriptionToTuples( subscription )
self._subscriptions.UpdateRow( i, display_tuple, data_tuple )
def Reset( self ):
message = '''Resetting these subscriptions will delete all their remembered urls, meaning when they next run, they will try to download them all over again. This may be expensive in time and data. Only do it if you are willing to wait. Do you want to do it?'''
with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:
if dlg.ShowModal() == wx.ID_YES:
for i in self._subscriptions.GetAllSelected():
subscription = self._subscriptions.GetClientData( i )[0]
subscription.Reset()
( display_tuple, data_tuple ) = self._ConvertSubscriptionToTuples( subscription )
self._subscriptions.UpdateRow( i, display_tuple, data_tuple )
def RetryFailures( self ):
for i in self._subscriptions.GetAllSelected():
subscription = self._subscriptions.GetClientData( i )[0]
seed_cache = subscription.GetSeedCache()
failed_seeds = seed_cache.GetSeeds( CC.STATUS_FAILED )
for seed in failed_seeds:
seed_cache.UpdateSeedStatus( seed, CC.STATUS_UNKNOWN )
( display_tuple, data_tuple ) = self._ConvertSubscriptionToTuples( subscription )
self._subscriptions.UpdateRow( i, display_tuple, data_tuple )
class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
@ -2848,7 +3305,7 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel ):
suggestions = []
suggestions.append( 'mangled parse/typo' )
suggestions.append( 'tag not applicable' )
suggestions.append( 'not applicable' )
suggestions.append( 'should be namespaced' )
with ClientGUIDialogs.DialogTextEntry( self, message, suggestions = suggestions ) as dlg:

25
include/ClientGUISerialisable.py
View File

@ -21,19 +21,23 @@ class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
self._title = wx.TextCtrl( self )
self._title.Bind( wx.EVT_TEXT, self.EventChanged )
self._payload_type = wx.TextCtrl( self )
self._payload_description = wx.TextCtrl( self )
self._text = wx.TextCtrl( self )
self._width = wx.SpinCtrl( self, min = 100, max = 4096 )
self._export = ClientGUICommon.BetterButton( self, 'export', self.Export )
#
( payload_type, payload_string ) = ClientSerialisable.GetPayloadTypeAndString( self._payload_obj )
( payload_description, payload_string ) = ClientSerialisable.GetPayloadDescriptionAndString( self._payload_obj )
self._payload_type.SetValue( payload_type )
self._payload_description.SetValue( payload_description )
self._payload_type.Disable()
self._payload_description.Disable()
self._width.SetValue( 512 )
self.EventChanged( None )
@ -43,8 +47,9 @@ class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
rows.append( ( 'export path: ', self._filepicker ) )
rows.append( ( 'title: ', self._title ) )
rows.append( ( 'payload type: ', self._payload_type ) )
rows.append( ( 'description (optional): ', self._text ) )
rows.append( ( 'payload description: ', self._payload_description ) )
rows.append( ( 'your description (optional): ', self._text ) )
rows.append( ( 'png width: ', self._width ) )
rows.append( ( '', self._export ) )
gridbox = ClientGUICommon.WrapInGrid( self, rows )
@ -84,7 +89,9 @@ class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
def Export( self ):
( payload_type, payload_string ) = ClientSerialisable.GetPayloadTypeAndString( self._payload_obj )
width = self._width.GetValue()
( payload_description, payload_string ) = ClientSerialisable.GetPayloadDescriptionAndString( self._payload_obj )
title = self._title.GetValue()
text = self._text.GetValue()
@ -95,10 +102,10 @@ class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
path += '.png'
ClientSerialisable.DumpToPng( payload_string, title, payload_type, text, path )
ClientSerialisable.DumpToPng( width, payload_string, title, payload_description, text, path )
self._export.SetLabelText( 'done!' )
wx.CallLater( 2000, self._export.SetLabelText, 'export' )

135
include/ClientImporting.py
View File

@ -1860,11 +1860,30 @@ class SeedCache( HydrusSerialisable.SerialisableBase ):
return result
def GetSeeds( self ):
def GetSeeds( self, status = None ):
with self._lock:
return list( self._seeds_ordered )
if status is None:
return list( self._seeds_ordered )
else:
seeds = []
for seed in self._seeds_ordered:
seed_info = self._seeds_to_info[ seed ]
if seed_info[ 'status' ] == status:
seeds.append( seed )
return seeds
@ -2020,7 +2039,9 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
self._gallery_stream_identifiers = ClientDownloading.GetGalleryStreamIdentifiers( self._gallery_identifier )
self._query = ''
( namespaces, search_value ) = ClientDefaults.GetDefaultNamespacesAndSearchValue( self._gallery_identifier )
self._query = search_value
self._period = 86400 * 7
self._get_tags_if_redundant = False
@ -2037,7 +2058,10 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
self._paused = False
self._import_file_options = ClientDefaults.GetDefaultImportFileOptions()
self._import_tag_options = ClientData.ImportTagOptions()
new_options = HydrusGlobals.client_controller.GetNewOptions()
self._import_tag_options = new_options.GetDefaultImportTagOptions( self._gallery_identifier )
self._last_checked = 0
self._last_error = 0
@ -2045,6 +2069,16 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
self._seed_cache = SeedCache()
def _GetErrorProhibitionTime( self ):
return self._last_error + HC.UPDATE_DURATION
def _GetNextPeriodicSyncTime( self ):
return self._last_checked + self._period
def _GetSerialisableInfo( self ):
serialisable_gallery_identifier = self._gallery_identifier.GetSerialisableTuple()
@ -2069,7 +2103,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
def _NoRecentErrors( self ):
return HydrusData.TimeHasPassed( self._last_error + HC.UPDATE_DURATION ) or self._check_now
return HydrusData.TimeHasPassed( self._GetErrorProhibitionTime() )
def _UpdateSerialisableInfo( self, version, old_serialisable_info ):
@ -2388,7 +2422,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
def _SyncQueryCanDoWork( self ):
return HydrusData.TimeHasPassed( self._last_checked + self._period ) or self._check_now
return HydrusData.TimeHasPassed( self._GetNextPeriodicSyncTime() ) or self._check_now
def CheckNow( self ):
@ -2401,25 +2435,6 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
return self._gallery_identifier
def GetLastCheckedText( self ):
periodic_next_check_time = self._last_checked + self._period
error_next_check_time = self._last_error + HC.UPDATE_DURATION
if error_next_check_time > periodic_next_check_time and not HydrusData.TimeHasPassed( error_next_check_time ):
interim_text = ' | due to error ' + HydrusData.ConvertTimestampToPrettySync( self._last_error ) + ', next check '
next_check_time = error_next_check_time
else:
interim_text = ' | next check '
next_check_time = periodic_next_check_time
return 'last checked ' + HydrusData.ConvertTimestampToPrettySync( self._last_checked ) + interim_text + HydrusData.ConvertTimestampToPrettyPending( next_check_time )
def GetQuery( self ):
return self._query
@ -2435,6 +2450,11 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
return self._seed_cache
def PauseResume( self ):
self._paused = not self._paused
def Reset( self ):
self._last_checked = 0
@ -2442,12 +2462,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
self._seed_cache = SeedCache()
def SetSeedCache( self, seed_cache ):
self._seed_cache = seed_cache
def SetTuple( self, gallery_identifier, gallery_stream_identifiers, query, period, get_tags_if_redundant, initial_file_limit, periodic_file_limit, paused, import_file_options, import_tag_options ):
def SetTuple( self, gallery_identifier, gallery_stream_identifiers, query, period, get_tags_if_redundant, initial_file_limit, periodic_file_limit, paused, import_file_options, import_tag_options, last_checked, last_error, check_now, seed_cache ):
self._gallery_identifier = gallery_identifier
self._gallery_stream_identifiers = gallery_stream_identifiers
@ -2461,16 +2476,22 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
self._import_file_options = import_file_options
self._import_tag_options = import_tag_options
self._last_checked = last_checked
self._last_error = last_error
self._check_now = check_now
self._seed_cache = seed_cache
def Sync( self ):
p1 = not self._paused
p2 = not HydrusGlobals.view_shutdown
p3 = self._NoRecentErrors()
p4 = self._SyncQueryCanDoWork()
p5 = self._WorkOnFilesCanDoWork()
p3 = self._check_now
p4 = self._NoRecentErrors()
p5 = self._SyncQueryCanDoWork()
p6 = self._WorkOnFilesCanDoWork()
if p1 and p2 and p3 and ( p4 or p5 ):
if p1 and p2 and ( p3 or p4 ) and ( p5 or p6 ):
job_key = ClientThreading.JobKey( pausable = False, cancellable = True )
@ -2520,9 +2541,51 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
def ToPrettyStrings( self ):
site = HC.site_type_string_lookup[ self._gallery_identifier.GetSiteType() ]
last_checked = HydrusData.ConvertTimestampToPrettySync( self._last_checked )
period = HydrusData.ConvertTimeDeltaToPrettyString( self._period )
error_next_check_time = self._last_error + HC.UPDATE_DURATION
if HydrusData.TimeHasPassed( error_next_check_time ):
error_text = ''
else:
error_text = 'yes'
urls = HydrusData.ConvertIntToPrettyString( self._seed_cache.GetSeedCount() )
failures = HydrusData.ConvertIntToPrettyString( self._seed_cache.GetSeedCount( CC.STATUS_FAILED ) )
if self._paused:
paused_text = 'yes'
else:
paused_text = ''
if self._check_now:
check_now_text = 'yes'
else:
check_now_text = ''
return ( self._name, site, period, last_checked, error_text, urls, failures, paused_text, check_now_text )
def ToTuple( self ):
return ( self._gallery_identifier, self._gallery_stream_identifiers, self._query, self._period, self._get_tags_if_redundant, self._initial_file_limit, self._periodic_file_limit, self._paused, self._import_file_options, self._import_tag_options )
return ( self._gallery_identifier, self._gallery_stream_identifiers, self._query, self._period, self._get_tags_if_redundant, self._initial_file_limit, self._periodic_file_limit, self._paused, self._import_file_options, self._import_tag_options, self._last_checked, self._last_error, self._check_now, self._seed_cache )
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_SUBSCRIPTION ] = Subscription

53
include/ClientNetworking.py
View File

@ -251,11 +251,11 @@ class HTTPConnectionManager( object ):
threading.Thread( target = self.DAEMONMaintainConnections, name = 'Maintain Connections' ).start()
def _DoRequest( self, method, location, path, query, request_headers, body, follow_redirects = True, report_hooks = None, temp_path = None, num_redirects_permitted = 4 ):
def _DoRequest( self, method, location, path, query, request_headers, body, follow_redirects = True, report_hooks = None, temp_path = None, hydrus_network = False, num_redirects_permitted = 4 ):
if report_hooks is None: report_hooks = []
connection = self._GetConnection( location )
connection = self._GetConnection( location, hydrus_network )
try:
@ -321,22 +321,22 @@ class HTTPConnectionManager( object ):
def _GetConnection( self, location ):
def _GetConnection( self, location, hydrus_network ):
with self._lock:
if location not in self._connections:
if ( location, hydrus_network ) not in self._connections:
connection = HTTPConnection( location )
connection = HTTPConnection( location, hydrus_network )
self._connections[ location ] = connection
self._connections[ ( location, hydrus_network ) ] = connection
return self._connections[ location ]
return self._connections[ ( location, hydrus_network ) ]
def Request( self, method, url, request_headers = None, body = '', return_everything = False, return_cookies = False, report_hooks = None, temp_path = None ):
def Request( self, method, url, request_headers = None, body = '', return_cookies = False, report_hooks = None, temp_path = None, hydrus_network = False ):
if request_headers is None: request_headers = {}
@ -344,9 +344,9 @@ class HTTPConnectionManager( object ):
follow_redirects = not return_cookies
( response, size_of_response, response_headers, cookies ) = self._DoRequest( method, location, path, query, request_headers, body, follow_redirects = follow_redirects, report_hooks = report_hooks, temp_path = temp_path )
( response, size_of_response, response_headers, cookies ) = self._DoRequest( method, location, path, query, request_headers, body, follow_redirects = follow_redirects, report_hooks = report_hooks, temp_path = temp_path, hydrus_network = hydrus_network )
if return_everything:
if hydrus_network:
return ( response, size_of_response, response_headers, cookies )
@ -377,13 +377,13 @@ class HTTPConnectionManager( object ):
connections_copy = dict( self._connections )
for ( location, connection ) in connections_copy.items():
for ( ( location, hydrus_network ), connection ) in connections_copy.items():
with connection.lock:
if connection.IsStale():
del self._connections[ location ]
del self._connections[ ( location, hydrus_network ) ]
@ -397,10 +397,12 @@ class HTTPConnectionManager( object ):
class HTTPConnection( object ):
def __init__( self, location ):
def __init__( self, location, hydrus_network ):
( self._scheme, self._host, self._port ) = location
self._hydrus_network = hydrus_network
self._timeout = 30
self.lock = threading.Lock()
@ -566,6 +568,13 @@ class HTTPConnection( object ):
if attempt_number <= 3:
if self._hydrus_network:
# we are talking to a new hydrus server, which uses https, and hence an http call gives badstatusline
self._scheme = 'https'
self._RefreshConnection()
return self._GetInitialResponse( method, path_and_query, request_headers, body, attempt_number = attempt_number + 1 )
@ -808,7 +817,23 @@ class HTTPConnection( object ):
def _RefreshConnection( self ):
if self._scheme == 'http': self._connection = httplib.HTTPConnection( self._host, self._port, timeout = self._timeout )
elif self._scheme == 'https': self._connection = httplib.HTTPSConnection( self._host, self._port, timeout = self._timeout )
elif self._scheme == 'https':
if self._hydrus_network:
# this negotiates decent encryption but won't check hostname or the certificate
context = ssl.SSLContext( ssl.PROTOCOL_SSLv23 )
context.options |= ssl.OP_NO_SSLv2
context.options |= ssl.OP_NO_SSLv3
self._connection = httplib.HTTPSConnection( self._host, self._port, timeout = self._timeout, context = context )
else:
self._connection = httplib.HTTPSConnection( self._host, self._port, timeout = self._timeout )
try:

60
include/ClientSerialisable.py
View File

@ -1,5 +1,6 @@
import ClientConstants as CC
import ClientImageHandling
import ClientImporting
import ClientParsing
import cv2
import HydrusConstants as HC
@ -25,10 +26,10 @@ png_font = cv2.FONT_HERSHEY_TRIPLEX
greyscale_text_color = 0
title_size = 0.7
payload_type_size = 0.5
payload_description_size = 0.5
text_size = 0.4
def CreateTopImage( width, title, payload_type, text ):
def CreateTopImage( width, title, payload_description, text ):
text_extent_bmp = wx.EmptyBitmap( 20, 20, 24 )
@ -38,9 +39,9 @@ def CreateTopImage( width, title, payload_type, text ):
basic_font_size = text_font.GetPointSize()
payload_type_font = wx.SystemSettings.GetFont( wx.SYS_DEFAULT_GUI_FONT )
payload_description_font = wx.SystemSettings.GetFont( wx.SYS_DEFAULT_GUI_FONT )
payload_type_font.SetPointSize( int( basic_font_size * 1.4 ) )
payload_description_font.SetPointSize( int( basic_font_size * 1.4 ) )
title_font = wx.SystemSettings.GetFont( wx.SYS_DEFAULT_GUI_FONT )
@ -49,8 +50,8 @@ def CreateTopImage( width, title, payload_type, text ):
dc.SetFont( text_font )
( gumpf, text_line_height ) = dc.GetTextExtent( 'abcdefghijklmnopqrstuvwxyz' )
dc.SetFont( payload_type_font )
( gumpf, payload_type_line_height ) = dc.GetTextExtent( 'abcdefghijklmnopqrstuvwxyz' )
dc.SetFont( payload_description_font )
( gumpf, payload_description_line_height ) = dc.GetTextExtent( 'abcdefghijklmnopqrstuvwxyz' )
dc.SetFont( title_font )
( gumpf, title_line_height ) = dc.GetTextExtent( 'abcdefghijklmnopqrstuvwxyz' )
@ -71,7 +72,7 @@ def CreateTopImage( width, title, payload_type, text ):
text_total_height += 6 # to bring the last 4 padding up to 10 padding
top_height = 10 + title_line_height + 10 + payload_type_line_height + 10 + text_total_height
top_height = 10 + title_line_height + 10 + payload_description_line_height + 10 + text_total_height
#
@ -99,11 +100,11 @@ def CreateTopImage( width, title, payload_type, text ):
current_y += t_height + 10
dc.SetFont( payload_type_font )
dc.SetFont( payload_description_font )
( t_width, t_height ) = dc.GetTextExtent( payload_type )
( t_width, t_height ) = dc.GetTextExtent( payload_description )
dc.DrawText( payload_type, ( width - t_width ) / 2, current_y )
dc.DrawText( payload_description, ( width - t_width ) / 2, current_y )
current_y += t_height + 10
@ -137,16 +138,12 @@ def CreateTopImage( width, title, payload_type, text ):
return top_image
def DumpToPng( payload, title, payload_type, text, path ):
def DumpToPng( width, payload, title, payload_description, text, path ):
payload_length = len( payload )
payload_string_length = payload_length + 4
square_width = int( float( payload_string_length ) ** 0.5 )
width = max( 512, square_width )
payload_height = int( float( payload_string_length ) / width )
if float( payload_string_length ) / width % 1.0 > 0:
@ -154,7 +151,7 @@ def DumpToPng( payload, title, payload_type, text, path ):
payload_height += 1
top_image = CreateTopImage( width, title, payload_type, text )
top_image = CreateTopImage( width, title, payload_description, text )
payload_length_header = struct.pack( '!I', payload_length )
@ -186,18 +183,31 @@ def DumpToPng( payload, title, payload_type, text, path ):
HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
def GetPayloadTypeAndString( payload_obj ):
def GetPayloadTypeString( payload_obj ):
if isinstance( payload_obj, HydrusSerialisable.SerialisableList ):
return 'A list of ' + HydrusData.ConvertIntToPrettyString( len( payload_obj ) ) + ' ' + GetPayloadTypeString( payload_obj[0] ) + 's'
else:
if isinstance( payload_obj, ClientParsing.ParseRootFileLookup ):
return 'File Lookup Script'
elif isinstance( payload_obj, ClientImporting.Subscription ):
return 'Subscription'
def GetPayloadDescriptionAndString( payload_obj ):
payload_string = payload_obj.DumpToNetworkString()
if isinstance( payload_obj, ClientParsing.ParseRootFileLookup ):
payload_obj_type_string = 'File Lookup Script'
payload_description = GetPayloadTypeString( payload_obj ) + ' - ' + HydrusData.ConvertIntToBytes( len( payload_string ) )
payload_type = payload_obj_type_string + ' - ' + HydrusData.ConvertIntToBytes( len( payload_string ) )
return ( payload_type, payload_string )
return ( payload_description, payload_string )
def LoadFromPng( path ):
@ -300,4 +310,4 @@ def WrapText( text, width, size, thickness ):
return lines

2
include/HydrusConstants.py
View File

@ -46,7 +46,7 @@ options = {}
# Misc
NETWORK_VERSION = 17
SOFTWARE_VERSION = 235
SOFTWARE_VERSION = 236
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )

11
include/HydrusController.py
View File

@ -181,7 +181,12 @@ class HydrusController( object ):
def GoodTimeToDoBackgroundWork( self ):
return not ( self.JustWokeFromSleep() or self.SystemBusy() )
return self.CurrentlyIdle() and not ( self.JustWokeFromSleep() or self.SystemBusy() )
def GoodTimeToDoForegroundWork( self ):
return True
def JustWokeFromSleep( self ):
@ -203,7 +208,7 @@ class HydrusController( object ):
self._daemons.append( HydrusThreading.DAEMONWorker( self, 'SleepCheck', HydrusDaemons.DAEMONSleepCheck, period = 120 ) )
self._daemons.append( HydrusThreading.DAEMONWorker( self, 'MaintainMemory', HydrusDaemons.DAEMONMaintainMemory, period = 300 ) )
self._daemons.append( HydrusThreading.DAEMONBigJobWorker( self, 'MaintainDB', HydrusDaemons.DAEMONMaintainDB, period = 300 ) )
self._daemons.append( HydrusThreading.DAEMONBackgroundWorker( self, 'MaintainDB', HydrusDaemons.DAEMONMaintainDB, period = 300 ) )
@ -347,4 +352,4 @@ class HydrusController( object ):
return self._Write( action, HC.LOW_PRIORITY, True, *args, **kwargs )

7
include/HydrusDaemons.py
View File

@ -9,10 +9,7 @@ import HydrusData
def DAEMONMaintainDB( controller ):
if controller.CurrentlyIdle():
controller.MaintainDB()
controller.MaintainDB()
def DAEMONMaintainMemory( controller ):
@ -21,4 +18,4 @@ def DAEMONMaintainMemory( controller ):
def DAEMONSleepCheck( controller ):
controller.SleepCheck()

21
include/HydrusData.py
View File

@ -313,21 +313,36 @@ def ConvertTimeDeltaToPrettyString( seconds ):
days = seconds / 86400
hours = ( seconds % 86400 ) / 3600
result = '%d' % days + ' days ' + '%d' % hours + ' hours'
result = '%d' % days + ' days'
if hours > 0:
result += ' %d' % hours + ' hours'
elif seconds > 3600:
hours = seconds / 3600
minutes = ( seconds % 3600 ) / 60
result = '%d' % hours + ' hours ' + '%d' % minutes + ' minutes'
result = '%d' % hours + ' hours'
if minutes > 0:
result += ' %d' % minutes + ' minutes'
else:
minutes = seconds / 60
seconds = seconds % 60
result = '%d' % minutes + ' minutes ' + '%d' % seconds + ' seconds'
result = '%d' % minutes + ' minutes'
if seconds > 0:
result += ' %d' % seconds + ' seconds'
elif seconds > 1:

219
include/HydrusEncryption.py
View File

@ -1,78 +1,54 @@
import Crypto.Cipher.AES
import Crypto.Cipher.PKCS1_OAEP
import Crypto.Hash.SHA256
import Crypto.Signature.PKCS1_v1_5
import Crypto.PublicKey.RSA
import hashlib
import HydrusConstants as HC
import HydrusGlobals
import os
import potr
import time
import traceback
import yaml
import zlib
import HydrusGlobals
def AESKeyToText( aes_key, iv ): return ( aes_key + iv ).encode( 'hex' )
AES_KEY_LENGTH = 32
AES_BLOCK_SIZE = 16
def AESTextToKey( text ):
def DecryptAES( aes_key, encrypted_message ):
try: keys = text.decode( 'hex' )
except: raise Exception( 'Could not understand that key!' )
aes_key = keys[:32]
iv = keys[32:]
return ( aes_key, iv )
def DecryptAES( aes_key, iv, encrypted_message ):
iv = encrypted_message[:AES_BLOCK_SIZE]
enciphered_message = encrypted_message[AES_BLOCK_SIZE:]
aes_cipher = Crypto.Cipher.AES.new( aes_key, Crypto.Cipher.AES.MODE_CFB, iv )
padded_message = aes_cipher.decrypt( encrypted_message )
padded_message = aes_cipher.decrypt( enciphered_message )
message = UnpadAES( padded_message )
return message
def DecryptAESFile( aes_key, iv, path ):
def DecryptAESStream( aes_key, stream_in, stream_out ):
iv = stream_in.read( AES_BLOCK_SIZE )
aes_cipher = Crypto.Cipher.AES.new( aes_key, Crypto.Cipher.AES.MODE_CFB, iv )
if '.encrypted' in path: path_to = path.replace( '.encrypted', '' )
else: path_to = path + '.decrypted'
next_block = stream_in.read( HC.READ_BLOCK_SIZE )
with open( path, 'rb' ) as encrypted_f:
while True:
with open( path_to, 'wb' ) as decrypted_f:
block = next_block
next_block = stream_in.read( HC.READ_BLOCK_SIZE )
decrypted_block = aes_cipher.decrypt( block )
if next_block == '':
next_block = encrypted_f.read( HC.READ_BLOCK_SIZE )
if next_block.startswith( 'hydrus encrypted zip' ): next_block = next_block.replace( 'hydrus encrypted zip', '', 1 )
while True:
block = next_block
next_block = encrypted_f.read( HC.READ_BLOCK_SIZE )
decrypted_block = aes_cipher.decrypt( block )
if len( next_block ) == 0:
decrypted_block = UnpadAES( decrypted_block )
decrypted_f.write( decrypted_block )
if len( next_block ) == 0: break
decrypted_block = UnpadAES( decrypted_block )
stream_out.write( decrypted_block )
if next_block == '':
break
return path_to
def DecryptPKCS( private_key, encrypted_message ):
@ -81,58 +57,54 @@ def DecryptPKCS( private_key, encrypted_message ):
message = rsa_cipher.decrypt( encrypted_message )
return message
def DeserialiseRSAKey( text ):
def EncryptAES( aes_key, iv, message ):
return Crypto.PublicKey.RSA.importKey( text )
def EncryptAES( aes_key, message ):
iv = GenerateIV()
padded_message = PadAES( message )
aes_cipher = Crypto.Cipher.AES.new( aes_key, Crypto.Cipher.AES.MODE_CFB, iv )
encrypted_message = aes_cipher.encrypt( padded_message )
enciphered_message = aes_cipher.encrypt( padded_message )
encrypted_message = iv + enciphered_message
return encrypted_message
def EncryptAESFile( path, preface = '' ):
def EncryptAESStream( aes_key, stream_in, stream_out ):
( aes_key, iv ) = GenerateAESKeyAndIV()
iv = GenerateIV()
stream_out.write( iv )
aes_cipher = Crypto.Cipher.AES.new( aes_key, Crypto.Cipher.AES.MODE_CFB, iv )
with open( path, 'rb' ) as decrypted_f:
with open( path + '.encrypted', 'wb' ) as encrypted_f:
encrypted_f.write( preface )
next_block = decrypted_f.read( HC.READ_BLOCK_SIZE )
while True:
block = next_block
next_block = decrypted_f.read( HC.READ_BLOCK_SIZE )
if len( next_block ) == 0:
# block must be the last block
block = PadAES( block )
encrypted_block = aes_cipher.encrypt( block )
encrypted_f.write( encrypted_block )
if len( next_block ) == 0: break
next_block = stream_in.read( HC.READ_BLOCK_SIZE )
aes_key_text = AESKeyToText( aes_key, iv )
with open( path + '.key', 'wb' ) as f:
while True:
f.write( aes_key_text )
block = next_block
next_block = stream_in.read( HC.READ_BLOCK_SIZE )
if next_block == '':
block = PadAES( block )
encrypted_block = aes_cipher.encrypt( block )
stream_out.write( encrypted_block )
if next_block == '':
break
def EncryptPKCS( public_key, message ):
@ -140,17 +112,17 @@ def EncryptPKCS( public_key, message ):
rsa_cipher = Crypto.Cipher.PKCS1_OAEP.new( public_key )
# my understanding is that I don't have to manually pad this, cause OAEP does it for me.
# if that is wrong, then lol
encrypted_message = rsa_cipher.encrypt( message )
return encrypted_message
def GenerateAESKeyAndIV():
def GenerateAESKey():
aes_key = os.urandom( 32 )
iv = os.urandom( 16 ) # initialisation vector, aes block_size is 16
return os.urandom( AES_KEY_LENGTH )
return ( aes_key, iv )
def GenerateIV():
return os.urandom( AES_BLOCK_SIZE )
def GenerateFilteredRandomBytes( byte_to_exclude, num_bytes ):
@ -165,21 +137,17 @@ def GenerateFilteredRandomBytes( byte_to_exclude, num_bytes ):
return ''.join( bytes )
def GenerateNewPrivateKey(): return Crypto.PublicKey.RSA.generate( 2048 ).exportKey()
def GetPublicKey( private_key_text ):
def GenerateRSAKeyPair():
private_key = TextToKey( private_key_text )
private_key = Crypto.PublicKey.RSA.generate( 2048 )
public_key = private_key.publickey()
return public_key.exportKey()
return ( private_key, public_key )
def TextToKey( text ): return Crypto.PublicKey.RSA.importKey( text )
def PadAES( message ):
block_size = 16
block_size = AES_BLOCK_SIZE
# get last byte
# add random gumpf (except for last byte), then add last byte again
@ -192,9 +160,13 @@ def PadAES( message ):
return message + pad
def SerialiseRSAKey( key ):
return key.exportKey()
def UnpadAES( message ):
block_size = 16
block_size = AES_BLOCK_SIZE
# check last byte, jump back to previous instance of that byte
@ -213,48 +185,3 @@ def UnpadAES( message ):
return message[:index_of_correct_end + 1]
# I based this on the excellent article by Darrik L Mazey, here:
# https://blog.darmasoft.net/2013/06/30/using-pure-python-otr.html
DEFAULT_POLICY_FLAGS = {}
DEFAULT_POLICY_FLAGS[ 'ALLOW_V1' ] = False
DEFAULT_POLICY_FLAGS[ 'ALLOW_V2' ] = True
DEFAULT_POLICY_FLAGS[ 'REQUIRE_ENCRYPTION' ] = True
GenerateOTRKey = potr.compatcrypto.generateDefaultKey
def LoadOTRKey( stream ): return potr.crypt.PK.parsePrivateKey( stream )[0]
def DumpOTRKey( key ): return key.serializePrivateKey()
class HydrusOTRContext( potr.context.Context ):
def getPolicy( self, key ):
if key in DEFAULT_POLICY_FLAGS: return DEFAULT_POLICY_FLAGS[ key ]
else: return False
def inject( self, msg, appdata = None ):
inject_catcher = appdata
inject_catcher.write( msg )
class HydrusOTRAccount( potr.context.Account ):
def __init__( self, name, privkey, trusts ):
potr.context.Account.__init__( self, name, 'hydrus network otr', 1024, privkey )
self.trusts = trusts
def saveTrusts( self ):
HydrusGlobals.controller.Write( 'otr_trusts', self.name, self.trusts )
# I need an accounts manager so there is only ever one copy of an account
# it should fetch name, privkey and trusts from db on bootup
# savettrusts should just spam to the db because it ain't needed that much.

203
include/HydrusRatingArchive.py Normal file
View File

@ -0,0 +1,203 @@
import os
import sqlite3
HASH_TYPE_MD5 = 0 # 16 bytes long
HASH_TYPE_SHA1 = 1 # 20 bytes long
HASH_TYPE_SHA256 = 2 # 32 bytes long
HASH_TYPE_SHA512 = 3 # 64 bytes long
# Please feel free to use this file however you wish.
# None of this is thread-safe, though, so don't try to do anything clever.
# A rating for hydrus is a float from 0.0 to 1.0
# dislike/like are 0.0 and 1.0
# numerical are fractions between 0.0 and 1.0
# for a four-star rating that allows 0 stars, the 5 possibles are: 0.0, 0.25, 0.5, 0.75, 1.0
# for a three-star rating that does not allow 0 stars, the three possibles are: 0.0, 0.5, 1.0
# in truth, at our level:
# a five-star rating that does allow stars is a six-star rating
# a ten-star rating that does not allow stars is a ten-star rating
# If you want to make a new rating archive for use in hydrus, you want to do something like:
# import HydrusRatingArchive
# hra = HydrusRatingArchive.HydrusRatingArchive( 'my_little_archive.db' )
# hra.SetHashType( HydrusRatingArchive.HASH_TYPE_MD5 )
# hra.SetNumberOfStars( 5 )
# hra.BeginBigJob()
# for ( hash, rating ) in my_rating_generator: hra.AddRating( hash, rating )
# hra.CommitBigJob()
# del hra
# If you are only adding a couple ratings, you can exclude the BigJob stuff. It just makes millions of sequential writes more efficient.
# Also, this manages hashes as bytes, not hex, so if you have something like:
# hash = ab156e87c5d6e215ab156e87c5d6e215
# Then go hash = hash.decode( 'hex' ) before you pass it to Add/Get/Has/SetRating
# And also feel free to contact me directly at hydrus.admin@gmail.com if you need help.
class HydrusRatingArchive( object ):
def __init__( self, path ):
self._path = path
if not os.path.exists( self._path ): create_db = True
else: create_db = False
self._InitDBCursor()
if create_db: self._InitDB()
def _InitDB( self ):
self._c.execute( 'CREATE TABLE hash_type ( hash_type INTEGER );', )
self._c.execute( 'CREATE TABLE number_of_stars ( number_of_stars INTEGER );', )
self._c.execute( 'CREATE TABLE ratings ( hash BLOB PRIMARY KEY, rating REAL );' )
def _InitDBCursor( self ):
self._db = sqlite3.connect( self._path, isolation_level = None, detect_types = sqlite3.PARSE_DECLTYPES )
self._c = self._db.cursor()
def BeginBigJob( self ): self._c.execute( 'BEGIN IMMEDIATE;' )
def CommitBigJob( self ):
self._c.execute( 'COMMIT;' )
self._c.execute( 'VACUUM;' )
def DeleteRating( self, hash ):
self._c.execute( 'DELETE FROM ratings WHERE hash = ?;', ( sqlite3.Binary( hash ), ) )
def GetHashType( self ):
result = self._c.execute( 'SELECT hash_type FROM hash_type;' ).fetchone()
if result is None:
result = self._c.execute( 'SELECT hash FROM hashes;' ).fetchone()
if result is None:
raise Exception( 'This archive has no hash type set, and as it has no files, no hash type guess can be made.' )
if len( hash ) == 16: hash_type = HASH_TYPE_MD5
elif len( hash ) == 20: hash_type = HASH_TYPE_SHA1
elif len( hash ) == 32: hash_type = HASH_TYPE_SHA256
elif len( hash ) == 64: hash_type = HASH_TYPE_SHA512
else:
raise Exception( 'This archive has non-standard hashes. Something is wrong.' )
self.SetHashType( hash_type )
return hash_type
else:
( hash_type, ) = result
return hash_type
def GetName( self ):
filename = os.path.basename( self._path )
if '.' in filename: filename = filename.split( '.', 1 )[0]
return filename
def GetNumberOfStars( self ):
result = self._c.execute( 'SELECT number_of_stars FROM number_of_stars;' ).fetchone()
if result is None:
raise Exception( 'This rating archive has no number of stars set.' )
else:
( number_of_stars, ) = result
return number_of_stars
def GetRating( self, hash ):
result = self._c.execute( 'SELECT rating FROM ratings WHERE hash = ?;', ( sqlite3.Binary( hash ), ) ).fetchone()
if result is None:
return None
else:
( rating, ) = result
return rating
def HasHash( self, hash ):
result = self._c.execute( 'SELECT 1 FROM ratings WHERE hash = ?;', ( sqlite3.Binary( hash ), ) ).fetchone()
if result is None:
return False
else:
return True
def IterateRatings( self ):
for row in self._c.execute( 'SELECT hash, rating FROM ratings;' ):
yield row
def SetHashType( self, hash_type ):
self._c.execute( 'DELETE FROM hash_type;' )
self._c.execute( 'INSERT INTO hash_type ( hash_type ) VALUES ( ? );', ( hash_type, ) )
def SetNumberOfStars( self, number_of_stars ):
self._c.execute( 'DELETE FROM number_of_stars;' )
self._c.execute( 'INSERT INTO number_of_stars ( number_of_stars ) VALUES ( ? );', ( number_of_stars, ) )
def SetRating( self, hash, rating ):
self._c.execute( 'REPLACE INTO ratings ( hash, rating ) VALUES ( ?, ? );', ( sqlite3.Binary( hash ), rating ) )

18
include/HydrusTags.py
View File

@ -71,8 +71,8 @@ def CensorshipMatch( tag, censorships ):
return False
def ConvertTagToSortable( t ):
if t[0].isdigit():
if t[0].isdecimal():
# We want to maintain that:
# 0 < 0a < 0b < 1 ( lexicographic comparison )
@ -86,7 +86,7 @@ def ConvertTagToSortable( t ):
for character in t:
if character.isdigit(): int_component += character
if character.isdecimal(): int_component += character
else: break
i += 1
@ -94,9 +94,15 @@ def ConvertTagToSortable( t ):
str_component = t[i:]
return ( int( int_component ), str_component )
number = int( int_component )
return ( number, str_component )
else:
return t
else: return t
def FilterNamespaces( tags, namespaces ):
@ -226,4 +232,4 @@ def RenderTag( tag ):
return tag

104
include/HydrusThreading.py
View File

@ -101,7 +101,10 @@ class DAEMONQueue( DAEMON ):
while self._queue.empty():
if IsThreadShuttingDown(): return
if IsThreadShuttingDown():
return
self._event.wait( self._period )
@ -129,7 +132,7 @@ class DAEMONQueue( DAEMON ):
class DAEMONWorker( DAEMON ):
def __init__( self, controller, name, callable, topics = None, period = 3600, init_wait = 3 ):
def __init__( self, controller, name, callable, topics = None, period = 3600, init_wait = 3, pre_call_wait = 0 ):
if topics is None: topics = []
@ -139,79 +142,52 @@ class DAEMONWorker( DAEMON ):
self._topics = topics
self._period = period
self._init_wait = init_wait
self._pre_call_wait = pre_call_wait
for topic in topics: self._controller.sub( self, 'set', topic )
self.start()
def _CanStart( self, time_started_waiting ):
return self._PreCallWaitIsDone( time_started_waiting ) and self._ControllerIsOKWithIt()
def _ControllerIsOKWithIt( self ):
return True
def _PreCallWaitIsDone( self, time_started_waiting ):
# just shave a bit off so things that don't have any wait won't somehow have to wait a single accidentaly cycle
time_to_start = ( float( time_started_waiting ) - 0.1 ) + self._pre_call_wait
return HydrusData.TimeHasPassed( time_to_start )
def run( self ):
self._event.wait( self._init_wait )
while True:
if IsThreadShuttingDown(): return
try:
self._callable( self._controller )
except HydrusExceptions.ShutdownException:
if IsThreadShuttingDown():
return
except Exception as e:
HydrusData.ShowText( 'Daemon ' + self._name + ' encountered an exception:' )
HydrusData.ShowException( e )
if IsThreadShuttingDown(): return
time_started_waiting = HydrusData.GetNow()
self._event.wait( self._period )
self._event.clear()
def set( self, *args, **kwargs ): self._event.set()
class DAEMONBigJobWorker( DAEMON ):
def __init__( self, controller, name, callable, topics = None, period = 3600, init_wait = 3, pre_callable_wait = 3 ):
if topics is None: topics = []
DAEMON.__init__( self, controller, name )
self._callable = callable
self._topics = topics
self._period = period
self._init_wait = init_wait
self._pre_callable_wait = pre_callable_wait
for topic in topics: self._controller.sub( self, 'set', topic )
self.start()
def run( self ):
self._event.wait( self._init_wait )
while True:
if IsThreadShuttingDown(): return
time_to_go = ( HydrusData.GetNow() - 1 ) + self._pre_callable_wait
while not ( HydrusData.TimeHasPassed( time_to_go ) and self._controller.GoodTimeToDoBackgroundWork() ):
while not self._CanStart( time_started_waiting ):
time.sleep( 1 )
if IsThreadShuttingDown(): return
if IsThreadShuttingDown():
return
try:
@ -239,6 +215,22 @@ class DAEMONBigJobWorker( DAEMON ):
def set( self, *args, **kwargs ): self._event.set()
# Big stuff like DB maintenance that we don't want to run while other important stuff is going on, like user interaction or vidya on another process
class DAEMONBackgroundWorker( DAEMONWorker ):
def _ControllerIsOKWithIt( self ):
return self._controller.GoodTimeToDoBackgroundWork()
# Big stuff that we want to run when the user sees, but not at the expense of something else, like laggy session load
class DAEMONForegroundWorker( DAEMONWorker ):
def _ControllerIsOKWithIt( self ):
return self._controller.GoodTimeToDoForegroundWork()
class THREADCallToThread( DAEMON ):
def __init__( self, controller ):
@ -301,4 +293,4 @@ class THREADCallToThread( DAEMON ):
time.sleep( 0.00001 )

51
include/HydrusVideoHandling.py
View File

@ -26,6 +26,57 @@ if not os.path.exists( FFMPEG_PATH ):
FFMPEG_PATH = os.path.basename( FFMPEG_PATH )
def GetFFMPEGVersion():
# open the file in a pipe, provoke an error, read output
cmd = [ FFMPEG_PATH, '-version' ]
try:
proc = subprocess.Popen( cmd, bufsize=10**5, stdout=subprocess.PIPE, stderr=subprocess.PIPE, startupinfo = HydrusData.GetSubprocessStartupInfo() )
except Exception as e:
if not os.path.exists( FFMPEG_PATH ):
return 'no ffmpeg found'
else:
HydrusData.ShowException( e )
return 'unable to execute ffmpeg'
infos = proc.stdout.read().decode( 'utf8' )
proc.terminate()
del proc
lines = infos.splitlines()
if len( lines ) > 0:
# typically 'ffmpeg version [VERSION] Copyright ...
top_line = lines[0]
if top_line.startswith( 'ffmpeg version ' ):
top_line = top_line.replace( 'ffmpeg version ', '' )
if ' ' in top_line:
version_string = top_line.split( ' ' )[0]
return version_string
return 'unknown'
def GetFFMPEGVideoProperties( path ):
info = Hydrusffmpeg_parse_infos( path )

7
include/ServerController.py
View File

@ -14,6 +14,7 @@ import ServerServer
import sys
import time
import traceback
import twisted.internet.ssl
from twisted.internet import reactor
from twisted.internet import defer
@ -197,6 +198,10 @@ class Controller( HydrusController.HydrusController ):
elif service_type == HC.TAG_REPOSITORY: service_object = ServerServer.HydrusServiceRepositoryTag( service_key, service_type, message )
elif service_type == HC.MESSAGE_DEPOT: return
#context_factory = twisted.internet.ssl.DefaultOpenSSLContextFactory( 'muh_key.key', 'muh_crt.crt' )
#self._services[ service_key ] = reactor.listenSSL( port, service_object, context_factory )
self._services[ service_key ] = reactor.listenTCP( port, service_object )
try:
@ -396,4 +401,4 @@ class Controller( HydrusController.HydrusController ):
self.CallToThread( do_it )

11
include/TestConstants.py
View File

@ -18,7 +18,7 @@ class FakeHTTPConnectionManager():
self._fake_responses = {}
def Request( self, method, url, request_headers = None, body = '', return_everything = False, return_cookies = False, report_hooks = None, temp_path = None ):
def Request( self, method, url, request_headers = None, body = '', return_cookies = False, report_hooks = None, temp_path = None, hydrus_network = False ):
if request_headers is None: request_headers = {}
if report_hooks is None: report_hooks = []
@ -32,11 +32,16 @@ class FakeHTTPConnectionManager():
response = 'path written to temporary path'
if return_everything: return ( response, size_of_response, response_headers, cookies )
if hydrus_network: return ( response, size_of_response, response_headers, cookies )
elif return_cookies: return ( response, cookies )
else: return response
def RequestHydrus( self, method, url, request_headers = None, body = '', report_hooks = None, temp_path = None ):
pass
def SetResponse( self, method, url, response, size_of_response = 100, response_headers = None, cookies = None ):
if response_headers is None: response_headers = {}
@ -48,4 +53,4 @@ class FakeHTTPConnectionManager():
class FakeWebSessionManager():
def GetCookies( self, *args, **kwargs ): return { 'session_cookie' : 'blah' }

131
include/TestHydrusEncryption.py
View File

@ -1,131 +0,0 @@
import ClientConstants as CC
import ClientGUIDialogs
import collections
import HydrusConstants as HC
import HydrusData
import HydrusEncryption
import os
import potr
import TestConstants
import unittest
import HydrusGlobals
class Catcher():
def __init__( self ):
self._last_write = ''
def GetLastWrite( self ):
l_w = self._last_write
self._last_write = ''
return l_w
def write( self, data ): self._last_write = data
class TestIM( unittest.TestCase ):
def test_otr( self ):
alice = HydrusData.GenerateKey().encode( 'hex' )
bob = HydrusData.GenerateKey().encode( 'hex' )
alice_privkey_hex = '0000000000808000000000000000944834d12b2ad788d34743102266aa9d87fc180577f977c2b201799a4149ca598819ff59591254cb312d1ad23d791a9355cd423c438cb0bc7000bb33377cf73be6fc900705c250d2bdba3287c8e545faf0653e44e66aefffda6e445947ff98cac7c02cb4911f9f527a6f25cf6b8aae4af2909b3c077b80bb00000014afb936c2487a867db906015d755f158e5bf38c1d00000080345d40c8fc329e254ef4be5efa7e1dc20484b982394d09fece366ef598db1a29f4b63160728de57058f405903ded01d6359242656f1e8c02a0b5c67f5d09496486f2f9f005abcec1470888bd7f31dbee8b0ce94b31ed36437dc2446b38829ba08927329bd1ecec0de1d2cd409f840ed2478cdf154a12f79815b29e75ea4a2e0f000000807731a186f55afcdebc34aba5a10130e5eafac0d0067c50f49be494a463271b34a657114c9b69c4fbe30302259feafe75f091b5c5670c7193e256bd7a5be2f3daee2d1a8bc4e04eec891cd6c4591edf40e5cbf8f3e1ca985a9b01d13768ea7160761af475b0097878376dbac6b1ce5b101fb1dd7da354e739791895caba52f14c000000146497dca1a62f1039a0ce8bfc99984de1cc5a9848'
bob_privkey_hex = '00000000008080000000000000741dae82c8c9a55a7f2a5eb9e4db0b3e5990de5df5d7e2a0dab221a8e1e8b92d99f70387458088215836ed1c42c157640578da801120aa50c180c7d9b4e72205b863ecbd6f43e2efbca04d4c6b1b184fd57bda231445ad4a5e9b7ada27ddd9b24c2cfdba77858e76072b5e87a0a4eb91608ffea42ded252bd700000014ec380fdb62ad0248746142c58654403f665c9701000000806e1aaee6b00ee1a77927b5c7a28089eb9bc147e7688091aeeff7de7c3fa98498748d0744f328c230991e9d8031b704d9fc2a87206d62e2f3b1c30b3a370a237368b04dbe826978a232666be84db52c398700d8e2dbc4f5cabc8bd1270f429ea54247a087fdedfac723bf8b1aa4cfad664646a51d97f96a7dffaef0c24d90a5f5000000803dff456298b4fdc4a08599790341f274c8ea7685101cd2d42fb90a34034f71ca0b9b1f2074ec41e1282bd6a3b74d855c82fcea411485da83f784ca15deb3b5372b544ae84fa6f9a8cd470bc8ebd8e60135098e4a4b608d2aea395b2053311f0802a6db0836e25170ce8e5670579f63445688113b93f8597e88d28f03c020c77800000014a762254ce091c8abf6acd0945e32436abbc1b3f2'
alice_privkey = HydrusEncryption.LoadOTRKey( alice_privkey_hex.decode( 'hex' ) )
bob_privkey = HydrusEncryption.LoadOTRKey( bob_privkey_hex.decode( 'hex' ) )
#alice_privkey = HydrusEncryption.GenerateOTRKey()
#bob_privkey = HydrusEncryption.GenerateOTRKey()
alice_account = HydrusEncryption.HydrusOTRAccount( alice, alice_privkey, {} )
bob_account = HydrusEncryption.HydrusOTRAccount( bob, bob_privkey, {} )
alice_context = HydrusEncryption.HydrusOTRContext( alice_account, bob )
bob_context = HydrusEncryption.HydrusOTRContext( bob_account, alice )
catcher = Catcher()
#
self.assertEqual( alice_context.state, potr.context.STATE_PLAINTEXT )
self.assertEqual( bob_context.state, potr.context.STATE_PLAINTEXT )
m = alice_context.sendMessage( potr.context.FRAGMENT_SEND_ALL, '' )
res = bob_context.receiveMessage( m, catcher )
m = catcher.GetLastWrite()
res = alice_context.receiveMessage( m, catcher )
m = catcher.GetLastWrite()
res = bob_context.receiveMessage( m, catcher )
m = catcher.GetLastWrite()
res = alice_context.receiveMessage( m, catcher )
m = catcher.GetLastWrite()
res = bob_context.receiveMessage( m, catcher )
self.assertEqual( alice_context.state, potr.context.STATE_ENCRYPTED )
self.assertEqual( bob_context.state, potr.context.STATE_ENCRYPTED )
self.assertEqual( bob_privkey.getPublicPayload(), alice_context.getCurrentKey().getPublicPayload() )
self.assertEqual( alice_privkey.getPublicPayload(), bob_context.getCurrentKey().getPublicPayload() )
#
self.assertEqual( alice_context.getCurrentTrust(), None )
alice_context.setCurrentTrust( 'verified' )
self.assertEqual( alice_context.getCurrentTrust(), 'verified' )
[ ( args, kwargs ) ] = HydrusGlobals.test_controller.GetWrite( 'otr_trusts' )
self.assertEqual( args, ( alice, { bob : { alice_context.getCurrentKey().cfingerprint() : 'verified' } } ) )
self.assertEqual( bob_context.getCurrentTrust(), None )
bob_context.setCurrentTrust( 'verified' )
self.assertEqual( bob_context.getCurrentTrust(), 'verified' )
[ ( args, kwargs ) ] = HydrusGlobals.test_controller.GetWrite( 'otr_trusts' )
self.assertEqual( args, ( bob, { alice : { bob_context.getCurrentKey().cfingerprint() : 'verified' } } ) )
#
m = alice_context.sendMessage( potr.context.FRAGMENT_SEND_ALL, 'hello bob', appdata = catcher )
m = catcher.GetLastWrite()
res = bob_context.receiveMessage( m, catcher )
( message, gumpf ) = res
self.assertEqual( message, 'hello bob' )
#
m = bob_context.sendMessage( potr.context.FRAGMENT_SEND_ALL, 'hello alice', appdata = catcher )
m = catcher.GetLastWrite()
res = alice_context.receiveMessage( m, catcher )
( message, gumpf ) = res
self.assertEqual( message, 'hello alice' )

BIN
static/check.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 537 B