RepoMirrors
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Version 136

This commit is contained in:
Hydrus 2014-11-12 17:33:13 -06:00
parent 426a70ad04
commit c9d89da975
21 changed files with 730 additions and 150 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
client.pyw
View File

@ -49,6 +49,7 @@ try:
print( traceback.format_exc() )
sys.stdout = initial_sys_stdout
sys.stderr = initial_sys_stderr

10
help/advanced.html
View File

@ -29,6 +29,16 @@
<p>Here is a particularly zoomed out view, after importing volume 2:</p>
<p><img src="gunnerkrigg_volume.png"/></p>
<p>Importing with tags is great for long-running series with well-formatted filenames, and will save you literally hours' finicky tagging.</p>
<h3>tag archives</h3>
<p>A user told me he had scraped all of danbooru's tags, and we got to discussing if his data could be integrated with hydrus.</p>
<p>There are some technical limitations that prohibit rawly importing that data into a hydrus tag repository (our different systems use different file hash standards), but I figured it would be possible to import his data into a hydrus <i>client</i>, at least for any locally stored files.</p>
<p>After a bit of planning, I developed the concept of a <i>tag archive</i>, which is just a big and efficiently indexed database of hashes and tags that you can share very easily. Storing a tag archive in the right directory in your client's database allows you to <i>synchronise</i> with that archive, importing whatever tags it thinks your files should have to whatever tag service you wish.</p>
<p>The folder to place archives is <i>install_dir/db/client_archives</i>. You can get some archives that I have prepared <a href="http://www.mediafire.com/hydrus">here</a>.</p>
<p>Once you have put something in your archive folder, start the client, and then go <i>services->manage services</i> and browse to your local tag service or a remote tag repository. You should see a box for managing tag archives.</p>
<p><img src="tag_archives.png" /></p>
<p>Tag archive sync works a bit like the advanced tag options when you download from a booru or other gallery service; you select the namespaces the archive offers that you are interested in, and then, when you click OK, the client will search the archive for your local files. It will fetch the archive's tags for those files, filter according to your namespace selection, and then add or pend the tags as appropriate, just as if you had entered them in the manage tags dialog.</p>
<p>New files that you import will also be checked against your archive syncs, keeping you up to date.</p>
<p>Be careful with this tool! If you have tens of thousands of files and sync with an archive that has tens of millions of mappings, you may end up adding or pending tens of thousands of tags. It may take several minutes to initialise the sync, as well. Make sure you get your namespaces set exactly how you want before you click OK on the dialog.</p>
<h3>export to zip and encrypted</h3>
<p>The export dialog now supports exporting to zip, and, further, encrypting it:</p>
<p><img src="export_zip.png" /></p>

20
help/changelog.html
View File

@ -8,6 +8,26 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 136</h3></li>
<ul>
<li>added tag archives</li>
<li>added tag archive sync initialisation on all existing local files</li>
<li>added tag archive sync maintenance on newly imported files</li>
<li>added a new db directory, client_archives</li>
<li>added tag archive sync options to tag services in manage services dialog</li>
<li>added local tags service to manage services dialog</li>
<li>added sha512 to local hash cache</li>
<li>added tag archive hash_type guessing</li>
<li>added a new dialog for selecting n arbitrary strings</li>
<li>got testing to work on all platforms</li>
<li>fixed the hydrus server for linux and os x; it now has a stay-alive frame rather than a taskbaricon</li>
<li>improved some dialog testing code so it would work on all platforms</li>
<li>fixed a deffered problem that was causing the server AMP test to hang on Linux and OS X</li>
<li>neatened and harmonised a bit of common file and network streaming code</li>
<li>improved some misc manage services dialog code</li>
<li>fixed a critical bug that was meaning certain service changes were not being saved to the database, so were being forgotten on restart</li>
<li>fixed some select-string code that wasn't taking sets of strings for a weird wx reason</li>
</ul>
<li><h3>version 135</h3></li>
<ul>
<li>added a menu option to any tag's right-click menu to open a new search page for that tag</li>

8
help/server.html
View File

@ -23,9 +23,9 @@
<li>Corollary: If you have a problem with something on some dude's server, please, <span class="warning">do not come to me</span>, as I can in no way help with your problem. If your ex-gf's nudes have leaked onto the internet, or you find something terribly offensive, or you just plain hate the free flow of information, I cannot help you at all. IT IS THE DAWN OF 20XX. WELCOME TO INTERNET</li>
</ul>
</li>
<li>This is not for noobs. If you are wholly unfamiliar with NAT (opening ports on firewalls) and setting up servers in general, this may be a step too far. If you get stuck, drag a nerd away from Xel'Naga Caverns to give you a hand.</li>
<li>If you want your services to thrive in a community, you shall have to manage them. If you throw out a hundred uploader keys and go on a long vacation, do not expect your long-crafted thirteen-point 'Superman vs Goku ONLY!!!!!' ruleset to be upheld. Your repositories will clog with unrelated rubbish and your bandwidth will disappear. Banning worthless people and pruning worthless content is an important part of any online ecosystem.</li>
<li>There are legal issues with running a file-hosting service. If you plan to do this on any kind of large scale, <span class="warning">especially</span> if you plan to sell access for cash-money, familiarise yourself with your legal rights and responsibilities. You may—without even meaning to—make yourself liable for others' sins.</li>
<li>This is not technically trivial. If you are wholly unfamiliar with browsing around your file system, or opening ports on firewalls, or setting up servers in general, this may be a step too far. If you get stuck, find a nerd to give you a hand.</li>
<li>If you want your community to thrive, you shall have to manage it. If you throw out a hundred uploader keys and go on a long vacation, do not expect your long-crafted thirteen-point 'Superman vs Goku ONLY!!!!!' ruleset to be upheld. Your repositories will clog with unrelated rubbish and your bandwidth will disappear.</li>
<li>If you plan to do this on any kind of large scale, familiarise yourself with your legal rights and responsibilities.</li>
</ul>
<h3>still keen?</h3>
<p><b><i>If this stuff be-fuzzles you, and you just want a simple server set up on the same computer you run the client, you can now go </i>help->i don't know what I am doing->just set up the server on this computer, please<i> and you _should_ be all set up with an admin service and tag/file repos automatically.</i></b></p>
@ -34,7 +34,7 @@
<li>A <b>server</b> is an instantiation of the executable process, server.exe. It has a complicated and flexible database that can run many different services in parallel.</li>
<li>A <b>service</b> sits on a port (e.g. 45871) and responds to certain http requests (e.g. /file or /update) that the hydrus client can plug into. A service might be a repository for a certain kind of data, the administration interface to manage what services run on a server, or anything else.</li>
</ul>
<p>Setting up a hydrus server is easy compared to, say, Apache. There are no .conf files to mess about with, and everything is controlled through the client. When started, the server does nothing more visible than placing an icon in your system tray. Right click that, and you only get an option to exit.</p>
<p>Setting up a hydrus server is easy compared to, say, Apache. There are no .conf files to mess about with, and everything is controlled through the client. When started, the server will place an icon in your system tray in Windows or open a small frame in Linux or OS X. To close the server, either right-click the system tray icon and select exit, or just close the frame.</p>
<p>The basic process for setting up a server is thus:</p>
<ul>
<li>Start the server.</li>

BIN
help/tag_archives.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

274
include/ClientDB.py
View File

@ -12,6 +12,7 @@ import HydrusImageHandling
import HydrusMessageHandling
import HydrusNATPunch
import HydrusServer
import HydrusTagArchive
import HydrusTags
import HydrusThreading
import ClientConstants as CC
@ -1356,6 +1357,11 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
if service_type in HC.TAG_SERVICES:
if 'tag_archive_sync' not in info: info[ 'tag_archive_sync' ] = {}
if service_type in HC.REPOSITORIES:
if 'first_timestamp' not in info: info[ 'first_timestamp' ] = None
@ -1434,6 +1440,7 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
shutil.copy( self._db_path, path + os.path.sep + 'client.db' )
if os.path.exists( self._db_path + '-wal' ): shutil.copy( self._db_path + '-wal', path + os.path.sep + 'client.db-wal' )
shutil.copytree( HC.CLIENT_ARCHIVES_DIR, path + os.path.sep + 'client_archives' )
shutil.copytree( HC.CLIENT_FILES_DIR, path + os.path.sep + 'client_files' )
shutil.copytree( HC.CLIENT_THUMBNAILS_DIR, path + os.path.sep + 'client_thumbnails' )
shutil.copytree( HC.CLIENT_UPDATES_DIR, path + os.path.sep + 'client_updates' )
@ -2810,6 +2817,46 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
return shutdown_timestamps
def _GetTagArchiveInfo( self, c ): return { archive_name : hta.GetNamespaces() for ( archive_name, hta ) in self._tag_archives.items() }
def _GetTagArchiveTags( self, c, hashes ):
result = {}
for ( archive_name, hta ) in self._tag_archives.items():
hash_type == hta.GetHashType()
sha256_to_archive_hashes = {}
if hash_type == HydrusTagArchive.HASH_TYPE_SHA256:
sha256_to_archive_hashes = { hash : hash for hash in hashes }
else:
if hash_type == HydrusTagArchive.HASH_TYPE_MD5: h = 'md5'
elif hash_type == HydrusTagArchive.HASH_TYPE_SHA1: h = 'sha1'
elif hash_type == HydrusTagArchive.HASH_TYPE_SHA512: h = 'sha512'
for hash in hashes:
hash_id = self._GetHashId( c, hash )
( archive_hash, ) = c.execute( 'SELECT ' + h + ' FROM local_hashes WHERE hash_id = ?;', ( hash_id, ) ).fetchone()
sha256_to_archive_hashes[ hash ] = archive_hash
hashes_to_tags = { hash : hta.GetMappings( sha256_to_archive_hashes[ hash ] ) for hash in hashes }
result[ archive_name ] = hashes_to_tags
return result
def _GetTagCensorship( self, c, service_key = None ):
if service_key is None:
@ -3095,9 +3142,9 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
self.pub_content_updates_after_commit( { HC.LOCAL_FILE_SERVICE_KEY : [ content_update ] } )
( md5, sha1 ) = HydrusFileHandling.GetMD5AndSHA1FromPath( path )
( md5, sha1, sha512 ) = HydrusFileHandling.GetExtraHashesFromPath( path )
c.execute( 'INSERT OR IGNORE INTO local_hashes ( hash_id, md5, sha1 ) VALUES ( ?, ?, ? );', ( hash_id, sqlite3.Binary( md5 ), sqlite3.Binary( sha1 ) ) )
c.execute( 'INSERT OR IGNORE INTO local_hashes ( hash_id, md5, sha1, sha512 ) VALUES ( ?, ?, ?, ? );', ( hash_id, sqlite3.Binary( md5 ), sqlite3.Binary( sha1 ), sqlite3.Binary( sha512 ) ) )
if not archive: self._InboxFiles( c, ( hash_id, ) )
@ -3119,6 +3166,22 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
self._ProcessContentUpdates( c, service_keys_to_content_updates )
tag_services = self._GetServices( c, HC.TAG_SERVICES )
for service in tag_services:
service_key = service.GetServiceKey()
info = service.GetInfo()
tag_archive_sync = info[ 'tag_archive_sync' ]
for ( archive_name, namespaces ) in tag_archive_sync.items():
try: self._SyncToTagArchive( c, hash_id, archive_name, namespaces, service_key )
except: pass
if generate_media_result:
if ( can_add or already_in_db ):
@ -3148,7 +3211,43 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
def _ProcessContentUpdates( self, c, service_keys_to_content_updates ):
def _InitialiseTagArchiveSync( self, c, archive_name, namespaces, service_key ):
prefix_string = 'initialising sync to tag archive ' + archive_name + ': '
job_key = HC.JobKey( pausable = False, cancellable = False )
message = HC.MessageGauge( HC.MESSAGE_TYPE_GAUGE, prefix_string + 'preparing', job_key )
HC.pubsub.pub( 'message', message )
hash_ids = [ hash_id for ( hash_id, ) in c.execute( 'SELECT hash_id FROM files_info WHERE service_id = ?;', ( self._local_file_service_id, ) ) ]
block_size = 100
next_block = []
for ( i, hash_id ) in enumerate( hash_ids ):
try: self._SyncToTagArchive( c, hash_id, archive_name, namespaces, service_key )
except: pass
if i % 100 == 0:
message.SetInfo( 'range', len( hash_ids ) )
message.SetInfo( 'value', i )
message.SetInfo( 'text', prefix_string + HC.ConvertIntToPrettyString( i ) + '/' + HC.ConvertIntToPrettyString( len( hash_ids ) ) )
message.SetInfo( 'mode', 'gauge' )
message.SetInfo( 'text', prefix_string + 'done!' )
message.SetInfo( 'mode', 'text' )
self.pub_after_commit( 'notify_new_pending' )
def _ProcessContentUpdates( self, c, service_keys_to_content_updates, pub_immediate = False ):
notify_new_downloads = False
notify_new_pending = False
@ -3574,17 +3673,25 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
if notify_new_downloads: self.pub_after_commit( 'notify_new_downloads' )
if notify_new_pending: self.pub_after_commit( 'notify_new_pending' )
if notify_new_parents: self.pub_after_commit( 'notify_new_parents' )
if notify_new_siblings:
if pub_immediate:
self.pub_after_commit( 'notify_new_siblings' )
self.pub_after_commit( 'notify_new_parents' )
HC.pubsub.pub( 'content_updates_data', service_keys_to_content_updates )
HC.pubsub.pub( 'content_updates_gui', service_keys_to_content_updates )
else:
if notify_new_downloads: self.pub_after_commit( 'notify_new_downloads' )
if notify_new_pending: self.pub_after_commit( 'notify_new_pending' )
if notify_new_parents: self.pub_after_commit( 'notify_new_parents' )
if notify_new_siblings:
self.pub_after_commit( 'notify_new_siblings' )
self.pub_after_commit( 'notify_new_parents' )
if notify_new_thumbnails: self.pub_after_commit( 'notify_new_thumbnails' )
self.pub_content_updates_after_commit( service_keys_to_content_updates )
if notify_new_thumbnails: self.pub_after_commit( 'notify_new_thumbnails' )
self.pub_content_updates_after_commit( service_keys_to_content_updates )
def _ProcessServiceUpdates( self, c, service_keys_to_service_updates ):
@ -3802,6 +3909,43 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
def _SyncToTagArchive( self, c, hash_id, archive_name, namespaces, service_key ):
hta = self._tag_archives[ archive_name ]
hash_type = hta.GetHashType()
hash = self._GetHash( c, hash_id )
if hash_type == HydrusTagArchive.HASH_TYPE_SHA256: archive_hash = hash
else:
if hash_type == HydrusTagArchive.HASH_TYPE_MD5: h = 'md5'
elif hash_type == HydrusTagArchive.HASH_TYPE_SHA1: h = 'sha1'
elif hash_type == HydrusTagArchive.HASH_TYPE_SHA512: h = 'sha512'
( archive_hash, ) = c.execute( 'SELECT ' + h + ' FROM local_hashes WHERE hash_id = ?;', ( hash_id, ) ).fetchone()
tags = hta.GetMappings( archive_hash )
desired_tags = HydrusTags.FilterNamespaces( tags, namespaces )
if len( desired_tags ) > 0:
if service_key == HC.LOCAL_TAG_SERVICE_KEY: action = HC.CONTENT_UPDATE_ADD
else: action = HC.CONTENT_UPDATE_PENDING
rows = [ ( tag, ( hash, ) ) for tag in desired_tags ]
content_updates = [ HC.ContentUpdate( HC.CONTENT_DATA_TYPE_MAPPINGS, action, row ) for row in rows ]
service_keys_to_content_updates = { service_key : content_updates }
self._ProcessContentUpdates( c, service_keys_to_content_updates, pub_immediate = True )
def _UpdateAutocompleteTagCacheFromFiles( self, c, file_service_id, hash_ids, direction ):
splayed_hash_ids = HC.SplayListForDB( hash_ids )
@ -4277,6 +4421,30 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
self.pub_after_commit( 'permissions_are_stale' )
if service_type in HC.TAG_SERVICES:
( old_info, ) = c.execute( 'SELECT info FROM services WHERE service_id = ?;', ( service_id, ) ).fetchone()
old_tag_archive_sync = old_info[ 'tag_archive_sync' ]
new_tag_archive_sync = info_update[ 'tag_archive_sync' ]
for archive_name in new_tag_archive_sync:
namespaces = set( new_tag_archive_sync[ archive_name ] )
if archive_name in old_tag_archive_sync:
old_namespaces = old_tag_archive_sync[ archive_name ]
namespaces.difference_update( old_namespaces )
if len( namespaces ) == 0: continue
self._InitialiseTagArchiveSync( c, archive_name, namespaces, service_key )
self._UpdateServiceInfo( c, service_id, info_update )
if service_type == HC.LOCAL_BOORU:
@ -4454,12 +4622,38 @@ class DB( ServiceDB ):
return site_id
def _InitArchives( self ):
self._tag_archives = {}
for filename in dircache.listdir( HC.CLIENT_ARCHIVES_DIR ):
if filename.endswith( '.db' ):
try:
hta = HydrusTagArchive.HydrusTagArchive( HC.CLIENT_ARCHIVES_DIR + os.path.sep + filename )
archive_name = filename[:-3]
self._tag_archives[ archive_name ] = hta
except Exception as e:
HC.ShowText( 'An archive failed to load on boot.' )
HC.ShowException( e )
def _InitDB( self ):
if not os.path.exists( self._db_path ):
HC.is_first_start = True
if not os.path.exists( HC.CLIENT_ARCHIVES_DIR ): os.mkdir( HC.CLIENT_ARCHIVES_DIR )
if not os.path.exists( HC.CLIENT_FILES_DIR ): os.mkdir( HC.CLIENT_FILES_DIR )
if not os.path.exists( HC.CLIENT_THUMBNAILS_DIR ): os.mkdir( HC.CLIENT_THUMBNAILS_DIR )
if not os.path.exists( HC.CLIENT_UPDATES_DIR ): os.mkdir( HC.CLIENT_UPDATES_DIR )
@ -4520,9 +4714,10 @@ class DB( ServiceDB ):
c.execute( 'CREATE TABLE hydrus_sessions ( service_id INTEGER PRIMARY KEY REFERENCES services ON DELETE CASCADE, session_key BLOB_BYTES, expiry INTEGER );' )
c.execute( 'CREATE TABLE local_hashes ( hash_id INTEGER PRIMARY KEY, md5 BLOB_BYTES, sha1 BLOB_BYTES );' )
c.execute( 'CREATE TABLE local_hashes ( hash_id INTEGER PRIMARY KEY, md5 BLOB_BYTES, sha1 BLOB_BYTES, sha512 BLOB_BYTES );' )
c.execute( 'CREATE INDEX local_hashes_md5_index ON local_hashes ( md5 );' )
c.execute( 'CREATE INDEX local_hashes_sha1_index ON local_hashes ( sha1 );' )
c.execute( 'CREATE INDEX local_hashes_sha512_index ON local_hashes ( sha512 );' )
c.execute( 'CREATE TABLE local_ratings ( service_id INTEGER REFERENCES services ON DELETE CASCADE, hash_id INTEGER, rating REAL, PRIMARY KEY( service_id, hash_id ) );' )
c.execute( 'CREATE INDEX local_ratings_hash_id_index ON local_ratings ( hash_id );' )
@ -5344,6 +5539,54 @@ class DB( ServiceDB ):
c.execute( 'UPDATE options SET options = ?;', ( HC.options, ) )
if version == 135:
if not os.path.exists( HC.CLIENT_ARCHIVES_DIR ): os.mkdir( HC.CLIENT_ARCHIVES_DIR )
#
extra_hashes_data = c.execute( 'SELECT * FROM local_hashes;' ).fetchall()
c.execute( 'DROP TABLE local_hashes;' )
c.execute( 'CREATE TABLE local_hashes ( hash_id INTEGER PRIMARY KEY, md5 BLOB_BYTES, sha1 BLOB_BYTES, sha512 BLOB_BYTES );' )
c.execute( 'CREATE INDEX local_hashes_md5_index ON local_hashes ( md5 );' )
c.execute( 'CREATE INDEX local_hashes_sha1_index ON local_hashes ( sha1 );' )
c.execute( 'CREATE INDEX local_hashes_sha512_index ON local_hashes ( sha512 );' )
for ( i, ( hash_id, md5, sha1 ) ) in enumerate( extra_hashes_data ):
hash = self._GetHash( c, hash_id )
try: path = CC.GetFilePath( hash )
except: continue
h_sha512 = hashlib.sha512()
with open( path, 'rb' ) as f:
for block in HC.ReadFileLikeAsBlocks( f, 65536 ): h_sha512.update( block )
sha512 = h_sha512.digest()
c.execute( 'INSERT INTO local_hashes ( hash_id, md5, sha1, sha512 ) VALUES ( ?, ?, ?, ? );', ( hash_id, sqlite3.Binary( md5 ), sqlite3.Binary( sha1 ), sqlite3.Binary( sha512 ) ) )
if i % 100 == 0: HC.pubsub.pub( 'set_splash_text', 'generating sha512 hashes: ' + HC.ConvertIntToPrettyString( i ) )
#
tag_service_info = c.execute( 'SELECT service_id, info FROM services WHERE service_type IN ' + HC.SplayListForDB( HC.TAG_SERVICES ) + ';' ).fetchall()
for ( service_id, info ) in tag_service_info:
info[ 'tag_archive_sync' ] = {}
c.execute( 'UPDATE services SET info = ? WHERE service_id = ?;', ( info, service_id ) )
c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
HC.is_db_updated = True
@ -5387,6 +5630,8 @@ class DB( ServiceDB ):
def ProcessRead( action, args, kwargs ):
if action == '4chan_pass': result = self._GetYAMLDump( c, YAML_DUMP_ID_SINGLE, '4chan_pass' )
elif action == 'tag_archive_info': result = self._GetTagArchiveInfo( c, *args, **kwargs )
elif action == 'tag_archive_tags': result = self._GetTagArchiveTags( c, *args, **kwargs )
elif action == 'autocomplete_contacts': result = self._GetAutocompleteContacts( c, *args, **kwargs )
elif action == 'autocomplete_tags': result = self._GetAutocompleteTags( c, *args, **kwargs )
elif action == 'contact_names': result = self._GetContactNames( c, *args, **kwargs )
@ -5539,6 +5784,8 @@ class DB( ServiceDB ):
HC.pubsub.pub( 'db_locked_status', '' )
self._InitArchives()
( db, c ) = self._GetDBCursor()
while not ( ( self._local_shutdown or HC.shutdown ) and self._jobs.empty() ):
@ -5611,6 +5858,7 @@ class DB( ServiceDB ):
shutil.copy( path + os.path.sep + 'client.db', self._db_path )
if os.path.exists( path + os.path.sep + 'client.db-wal' ): shutil.copy( path + os.path.sep + 'client.db-wal', self._db_path + '-wal' )
shutil.copytree( path + os.path.sep + 'client_archives', HC.CLIENT_ARCHIVES_DIR )
shutil.copytree( path + os.path.sep + 'client_files', HC.CLIENT_FILES_DIR )
shutil.copytree( path + os.path.sep + 'client_thumbnails', HC.CLIENT_THUMBNAILS_DIR )
shutil.copytree( path + os.path.sep + 'client_updates', HC.CLIENT_UPDATES_DIR )

2
include/ClientGUICommon.py
View File

@ -968,7 +968,7 @@ class BetterChoice( wx.Choice ):
selection = self.GetSelection()
if selection != wx.NOT_FOUND: return self.GetClientData( selection )
else: raise Exception( 'choice not chosen' )
else: raise Exception( 'Choice not chosen!' )
def SelectClientData( self, client_data ):

60
include/ClientGUIDialogs.py
View File

@ -4856,6 +4856,62 @@ class DialogSelectImageboard( Dialog ):
def GetImageboard( self ): return self._tree.GetItemData( self._tree.GetSelection() ).GetData()
class DialogCheckFromListOfStrings( Dialog ):
def __init__( self, parent, title, list_of_strings, checked_strings = [] ):
def InitialiseControls():
self._strings = wx.CheckListBox( self )
self._ok = wx.Button( self, id = wx.ID_OK, label = 'ok' )
self._cancel = wx.Button( self, id = wx.ID_CANCEL, label = 'cancel' )
def PopulateControls():
for s in list_of_strings: self._strings.Append( s )
for s in checked_strings:
i = self._strings.FindString( s )
if i != wx.NOT_FOUND: self._strings.Check( i, True )
def ArrangeControls():
hbox = wx.BoxSizer( wx.HORIZONTAL )
hbox.AddF( self._ok, FLAGS_MIXED )
hbox.AddF( self._cancel, FLAGS_MIXED )
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.AddF( self._strings, FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( hbox, FLAGS_BUTTON_SIZERS )
self.SetSizer( vbox )
( x, y ) = self.GetEffectiveMinSize()
if x < 320: x = 320
self.SetInitialSize( ( x, y ) )
Dialog.__init__( self, parent, title )
InitialiseControls()
PopulateControls()
ArrangeControls()
def GetChecked( self ): return self._strings.GetCheckedStrings()
class DialogSelectFromListOfStrings( Dialog ):
def __init__( self, parent, title, list_of_strings ):
@ -4864,7 +4920,7 @@ class DialogSelectFromListOfStrings( Dialog ):
self._hidden_cancel = wx.Button( self, id = wx.ID_CANCEL, size = ( 0, 0 ) )
self._strings = wx.ListBox( self, choices = list_of_strings )
self._strings = wx.ListBox( self )
self._strings.Bind( wx.EVT_KEY_DOWN, self.EventKeyDown )
self._strings.Bind( wx.EVT_LISTBOX_DCLICK, self.EventSelect )
@ -4875,7 +4931,7 @@ class DialogSelectFromListOfStrings( Dialog ):
def PopulateControls():
pass
for s in list_of_strings: self._strings.Append( s )
def ArrangeControls():

274
include/ClientGUIDialogsManage.py
View File

@ -4764,13 +4764,14 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
def PopulateControls():
manageable_service_types = HC.RESTRICTED_SERVICES + [ HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL, HC.LOCAL_BOORU ]
manageable_service_types = HC.RESTRICTED_SERVICES + [ HC.LOCAL_TAG, HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL, HC.LOCAL_BOORU ]
for service_type in manageable_service_types:
if service_type == HC.LOCAL_RATING_LIKE: name = 'like/dislike ratings'
elif service_type == HC.LOCAL_RATING_NUMERICAL: name = 'numerical ratings'
elif service_type == HC.LOCAL_BOORU: name = 'booru'
elif service_type == HC.LOCAL_TAG: name = 'local tags'
elif service_type == HC.TAG_REPOSITORY: name = 'tag repositories'
elif service_type == HC.FILE_REPOSITORY: name = 'file repositories'
#elif service_type == HC.MESSAGE_DEPOT: name = 'message repositories'
@ -5106,7 +5107,7 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
service_type = self._listbooks_to_service_types[ services_listbook ]
if service_type == HC.LOCAL_BOORU:
if service_type in HC.NONEDITABLE_SERVICES:
self._add.Disable()
self._remove.Disable()
@ -5187,27 +5188,30 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
def InitialiseControls():
if service_type in HC.REMOTE_SERVICES: title = 'name and credentials'
else: title = 'name'
self._credentials_panel = ClientGUICommon.StaticBox( self, title )
self._service_name = wx.TextCtrl( self._credentials_panel )
if service_type in HC.REMOTE_SERVICES:
if service_type not in HC.NONEDITABLE_SERVICES:
host = info[ 'host' ]
port = info[ 'port' ]
if service_type in HC.REMOTE_SERVICES: title = 'name and credentials'
else: title = 'name'
if 'access_key' in info: access_key = info[ 'access_key' ]
else: access_key = None
self._credentials_panel = ClientGUICommon.StaticBox( self, title )
credentials = CC.Credentials( host, port, access_key )
self._service_name = wx.TextCtrl( self._credentials_panel )
self._service_credentials = wx.TextCtrl( self._credentials_panel, value = credentials.GetConnectionString() )
self._check_service = wx.Button( self._credentials_panel, label = 'test credentials' )
self._check_service.Bind( wx.EVT_BUTTON, self.EventCheckService )
if service_type in HC.REMOTE_SERVICES:
host = info[ 'host' ]
port = info[ 'port' ]
if 'access_key' in info: access_key = info[ 'access_key' ]
else: access_key = None
credentials = CC.Credentials( host, port, access_key )
self._service_credentials = wx.TextCtrl( self._credentials_panel, value = credentials.GetConnectionString() )
self._check_service = wx.Button( self._credentials_panel, label = 'test credentials' )
self._check_service.Bind( wx.EVT_BUTTON, self.EventCheckService )
if service_type in HC.REPOSITORIES:
@ -5220,7 +5224,7 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
self._reset.Bind( wx.EVT_BUTTON, self.EventServiceReset )
if service_type in ( HC.LOCAL_RATING_LIKE, HC.LOCAL_RATING_NUMERICAL ):
if service_type in HC.RATINGS_SERVICES:
self._local_rating_panel = ClientGUICommon.StaticBox( self, 'local rating configuration' )
@ -5241,6 +5245,26 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
self._lower.SetValue( lower )
self._upper = wx.SpinCtrl( self._local_rating_panel, min = -2000, max = 2000 )
self._upper.SetValue( upper )
if service_type in HC.TAG_SERVICES:
self._archive_info = HC.app.Read( 'tag_archive_info' )
self._archive_panel = ClientGUICommon.StaticBox( self, 'archive synchronisation' )
self._archive_sync = wx.ListBox( self._archive_panel, size = ( -1, 100 ) )
self._archive_sync_add = wx.Button( self._archive_panel, label = 'add' )
self._archive_sync_add.Bind( wx.EVT_BUTTON, self.EventArchiveAdd )
self._archive_sync_edit = wx.Button( self._archive_panel, label = 'edit' )
self._archive_sync_edit.Bind( wx.EVT_BUTTON, self.EventArchiveEdit )
self._archive_sync_remove = wx.Button( self._archive_panel, label = 'remove' )
self._archive_sync_remove.Bind( wx.EVT_BUTTON, self.EventArchiveRemove )
if service_type == HC.LOCAL_BOORU:
@ -5257,13 +5281,30 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
def PopulateControls():
self._service_name.SetValue( name )
if service_type not in HC.NONEDITABLE_SERVICES:
self._service_name.SetValue( name )
if service_type in HC.REPOSITORIES:
self._pause_synchronisation.SetValue( info[ 'paused' ] )
if service_type in HC.TAG_SERVICES:
for ( archive_name, namespaces ) in info[ 'tag_archive_sync' ].items():
name_to_display = self._GetArchiveNameToDisplay( archive_name, namespaces )
self._archive_sync.Append( name_to_display, ( archive_name, namespaces ) )
potential_archives = self._GetPotentialArchives()
if len( potential_archives ) == 0: self._archive_sync_add.Disable()
if service_type == HC.LOCAL_BOORU:
self._port.SetValue( info[ 'port' ] )
@ -5278,28 +5319,31 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
vbox = wx.BoxSizer( wx.VERTICAL )
if service_type == HC.LOCAL_BOORU: self._credentials_panel.Hide()
gridbox = wx.FlexGridSizer( 0, 2 )
gridbox.AddGrowableCol( 1, 1 )
gridbox.AddF( wx.StaticText( self._credentials_panel, label = 'name' ), FLAGS_MIXED )
gridbox.AddF( self._service_name, FLAGS_EXPAND_BOTH_WAYS )
if service_type in HC.REMOTE_SERVICES:
if service_type not in HC.NONEDITABLE_SERVICES:
gridbox.AddF( wx.StaticText( self._credentials_panel, label = 'credentials' ), FLAGS_MIXED )
gridbox.AddF( self._service_credentials, FLAGS_EXPAND_BOTH_WAYS )
self._credentials_panel.Hide()
gridbox.AddF( ( 20, 20 ), FLAGS_MIXED )
gridbox.AddF( self._check_service, FLAGS_LONE_BUTTON )
gridbox = wx.FlexGridSizer( 0, 2 )
gridbox.AddGrowableCol( 1, 1 )
gridbox.AddF( wx.StaticText( self._credentials_panel, label = 'name' ), FLAGS_MIXED )
gridbox.AddF( self._service_name, FLAGS_EXPAND_BOTH_WAYS )
if service_type in HC.REMOTE_SERVICES:
gridbox.AddF( wx.StaticText( self._credentials_panel, label = 'credentials' ), FLAGS_MIXED )
gridbox.AddF( self._service_credentials, FLAGS_EXPAND_BOTH_WAYS )
gridbox.AddF( ( 20, 20 ), FLAGS_MIXED )
gridbox.AddF( self._check_service, FLAGS_LONE_BUTTON )
self._credentials_panel.AddF( gridbox, FLAGS_EXPAND_SIZER_BOTH_WAYS )
vbox.AddF( self._credentials_panel, FLAGS_EXPAND_PERPENDICULAR )
self._credentials_panel.AddF( gridbox, FLAGS_EXPAND_SIZER_BOTH_WAYS )
vbox.AddF( self._credentials_panel, FLAGS_EXPAND_PERPENDICULAR )
if service_type in HC.REPOSITORIES:
@ -5337,6 +5381,20 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
vbox.AddF( self._local_rating_panel, FLAGS_EXPAND_PERPENDICULAR )
if service_type in HC.TAG_SERVICES:
hbox = wx.BoxSizer( wx.HORIZONTAL )
hbox.AddF( self._archive_sync_add, FLAGS_MIXED )
hbox.AddF( self._archive_sync_edit, FLAGS_MIXED )
hbox.AddF( self._archive_sync_remove, FLAGS_MIXED )
self._archive_panel.AddF( self._archive_sync, FLAGS_EXPAND_BOTH_WAYS )
self._archive_panel.AddF( hbox, FLAGS_BUTTON_SIZERS )
vbox.AddF( self._archive_panel, FLAGS_EXPAND_PERPENDICULAR )
if service_type == HC.LOCAL_BOORU:
hbox = wx.BoxSizer( wx.HORIZONTAL )
@ -5361,6 +5419,119 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
ArrangeControls()
def _GetArchiveNameToDisplay( self, archive_name, namespaces ):
if len( namespaces ) == 0: name_to_display = archive_name
else: name_to_display = archive_name + ' (' + ', '.join( HC.ConvertUglyNamespacesToPrettyStrings( namespaces ) ) + ')'
return name_to_display
def _GetPotentialArchives( self ):
existing_syncs = set()
for i in range( self._archive_sync.GetCount() ):
( archive_name, namespaces ) = self._archive_sync.GetClientData( i )
existing_syncs.add( archive_name )
potential_archives = { archive_name for archive_name in self._archive_info.keys() if archive_name not in existing_syncs }
return potential_archives
def EventArchiveAdd( self, event ):
if self._archive_sync.GetCount() == 0:
wx.MessageBox( 'Be careful with this tool! Synching a lot of files to a large archive can take a very long time to initialise.' )
potential_archives = self._GetPotentialArchives()
if len( potential_archives ) == 1:
( archive_name, ) = potential_archives
wx.MessageBox( 'There is only one tag archive, ' + archive_name + ', to select, so I am selecting it for you.' )
else:
with ClientGUIDialogs.DialogSelectFromListOfStrings( self, 'Select the tag archive to add', potential_archives ) as dlg:
if dlg.ShowModal() == wx.ID_OK: archive_name = dlg.GetString()
else: return
potential_namespaces = self._archive_info[ archive_name ]
with ClientGUIDialogs.DialogCheckFromListOfStrings( self, 'Select namespaces', HC.ConvertUglyNamespacesToPrettyStrings( potential_namespaces ) ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
namespaces = HC.ConvertPrettyStringsToUglyNamespaces( dlg.GetChecked() )
else: return
name_to_display = self._GetArchiveNameToDisplay( archive_name, namespaces )
self._archive_sync.Append( name_to_display, ( archive_name, namespaces ) )
potential_archives = self._GetPotentialArchives()
if len( potential_archives ) == 0: self._archive_sync_add.Disable()
def EventArchiveEdit( self, event ):
selection = self._archive_sync.GetSelection()
if selection != wx.NOT_FOUND:
( archive_name, existing_namespaces ) = self._archive_sync.GetClientData( selection )
if archive_name not in self._archive_info.keys():
wx.MessageBox( 'This archive does not seem to exist any longer!' )
return
archive_namespaces = self._archive_info[ archive_name ]
with ClientGUIDialogs.DialogCheckFromListOfStrings( self, 'Select namespaces', HC.ConvertUglyNamespacesToPrettyStrings( archive_namespaces ), HC.ConvertUglyNamespacesToPrettyStrings( existing_namespaces ) ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
namespaces = HC.ConvertPrettyStringsToUglyNamespaces( dlg.GetChecked() )
else: return
name_to_display = self._GetArchiveNameToDisplay( archive_name, namespaces )
self._archive_sync.SetString( selection, name_to_display )
self._archive_sync.SetClientData( selection, ( archive_name, namespaces ) )
def EventArchiveRemove( self, event ):
selection = self._archive_sync.GetSelection()
if selection != wx.NOT_FOUND: self._archive_sync.Delete( selection )
potential_archives = self._GetPotentialArchives()
if len( potential_archives ) == 0: self._archive_sync_add.Disable()
else: self._archive_sync_add.Enable()
def EventCheckService( self, event ):
( service_key, service_type, name, info ) = self.GetInfo()
@ -5407,9 +5578,14 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
( service_key, service_type, name, info ) = self._original_info
name = self._service_name.GetValue()
info = dict( info )
if name == '': raise Exception( 'Please enter a name' )
if service_type not in HC.NONEDITABLE_SERVICES:
name = self._service_name.GetValue()
if name == '': raise Exception( 'Please enter a name' )
if service_type in HC.REMOTE_SERVICES:
@ -5462,6 +5638,20 @@ class DialogManageServices( ClientGUIDialogs.Dialog ):
info[ 'upper' ] = upper
if service_type in HC.TAG_SERVICES:
tag_archives = {}
for i in range( self._archive_sync.GetCount() ):
( archive_name, namespaces ) = self._archive_sync.GetClientData( i )
tag_archives[ archive_name ] = namespaces
info[ 'tag_archive_sync' ] = tag_archives
if service_type == HC.LOCAL_BOORU:
info[ 'port' ] = self._port.GetValue()

34
include/HydrusConstants.py
View File

@ -7,6 +7,7 @@ BASE_DIR = sys.path[0]
BIN_DIR = BASE_DIR + os.path.sep + 'bin'
DB_DIR = BASE_DIR + os.path.sep + 'db'
CLIENT_ARCHIVES_DIR = DB_DIR + os.path.sep + 'client_archives'
CLIENT_FILES_DIR = DB_DIR + os.path.sep + 'client_files'
SERVER_FILES_DIR = DB_DIR + os.path.sep + 'server_files'
CLIENT_THUMBNAILS_DIR = DB_DIR + os.path.sep + 'client_thumbnails'
@ -64,7 +65,7 @@ options = {}
# Misc
NETWORK_VERSION = 15
SOFTWARE_VERSION = 135
SOFTWARE_VERSION = 136
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )
@ -193,7 +194,9 @@ RATINGS_SERVICES = [ LOCAL_RATING_LIKE, LOCAL_RATING_NUMERICAL, RATING_LIKE_REPO
REPOSITORIES = [ TAG_REPOSITORY, FILE_REPOSITORY, RATING_LIKE_REPOSITORY, RATING_NUMERICAL_REPOSITORY ]
RESTRICTED_SERVICES = ( REPOSITORIES ) + [ SERVER_ADMIN, MESSAGE_DEPOT ]
REMOTE_SERVICES = list( RESTRICTED_SERVICES )
TAG_SERVICES = [ LOCAL_TAG, TAG_REPOSITORY ]
LOCAL_SERVICES = [ LOCAL_FILE, LOCAL_TAG, LOCAL_RATING_LIKE, LOCAL_RATING_NUMERICAL, LOCAL_BOORU, COMBINED_FILE, COMBINED_TAG ]
NONEDITABLE_SERVICES = [ LOCAL_BOORU, LOCAL_FILE, LOCAL_TAG ]
ALL_SERVICES = list( REMOTE_SERVICES ) + list( LOCAL_SERVICES )
SERVICES_WITH_THUMBNAILS = [ FILE_REPOSITORY, LOCAL_FILE ]
@ -777,6 +780,14 @@ def ConvertPortablePathToAbsPath( portable_path ):
if os.path.exists( abs_path ): return abs_path
else: return None
def ConvertPrettyStringsToUglyNamespaces( pretty_strings ):
result = { s for s in pretty_strings if s != 'no namespace' }
if 'no namespace' in pretty_strings: result.add( '' )
return result
def ConvertServiceKeysToContentUpdatesToPrettyString( service_keys_to_content_updates ):
num_files = 0
@ -1071,6 +1082,16 @@ def ConvertTimeToPrettyTime( secs ):
return time.strftime( '%H:%M:%S', time.gmtime( secs ) )
def ConvertUglyNamespacesToPrettyStrings( namespaces ):
result = [ namespace for namespace in namespaces if namespace != '' and namespace is not None ]
result.sort()
if '' in namespaces or None in namespaces: result.insert( 0, 'no namespace' )
return result
def ConvertUnitToInteger( unit ):
if unit == 'B': return 1
@ -1179,6 +1200,17 @@ def MergeKeyToListDicts( key_to_list_dicts ):
return result
def ReadFileLikeAsBlocks( f, block_size ):
next_block = f.read( block_size )
while next_block != '':
yield next_block
next_block = f.read( block_size )
def SearchEntryMatchesPredicate( search_entry, predicate ):
( predicate_type, info ) = predicate.GetInfo()

1
include/HydrusDownloading.py
View File

@ -6,7 +6,6 @@ import HydrusExceptions
import HydrusNetworking
import HydrusThreading
import json
import lxml
import os
import pafy
import threading

42
include/HydrusFileHandling.py
View File

@ -131,46 +131,34 @@ def GetHashFromPath( path ):
h = hashlib.sha256()
block_size = 65536
with open( path, 'rb' ) as f:
next_block = f.read( 65536 )
while next_block != '':
h.update( next_block )
next_block = f.read( 65536 )
return h.digest()
for block in HC.ReadFileLikeAsBlocks( f, 65536 ): h.update( block )
def GetMD5AndSHA1FromPath( path ):
return h.digest()
def GetExtraHashesFromPath( path ):
h_md5 = hashlib.md5()
h_sha1 = hashlib.sha1()
block_size = 65536
h_sha512 = hashlib.sha512()
with open( path, 'rb' ) as f:
next_block = f.read( 65536 )
while next_block != '':
for block in HC.ReadFileLikeAsBlocks( f, 65536 ):
h_md5.update( next_block )
h_sha1.update( next_block )
next_block = f.read( 65536 )
h_md5.update( block )
h_sha1.update( block )
h_sha512.update( block )
md5 = h_md5.digest()
sha1 = h_sha1.digest()
return ( md5, sha1 )
md5 = h_md5.digest()
sha1 = h_sha1.digest()
sha512 = h_sha512.digest()
return ( md5, sha1, sha512 )
header_and_mime = [
( 0, '\xff\xd8', HC.IMAGE_JPEG ),

18
include/HydrusNetworking.py
View File

@ -286,13 +286,11 @@ class HTTPConnection( object ):
data = ''
next_block = response.read( self.read_block_size )
while next_block != '':
for block in HC.ReadFileLikeAsBlocks( response, self.read_block_size ):
if HC.shutdown: raise Exception( 'Application is shutting down!' )
data += next_block
data += block
if content_length is not None and len( data ) > content_length:
@ -301,8 +299,6 @@ class HTTPConnection( object ):
for hook in report_hooks: hook( content_length, len( data ) )
next_block = response.read( self.read_block_size )
size_of_response = len( data )
@ -360,25 +356,21 @@ class HTTPConnection( object ):
with open( temp_path, 'wb' ) as f:
next_block = response.read( self.read_block_size )
while next_block != '':
for block in HC.ReadFileLikeAsBlocks( response, self.read_block_size ):
if HC.shutdown: raise Exception( 'Application is shutting down!' )
size_of_response += len( next_block )
size_of_response += len( block )
if content_length is not None and size_of_response > content_length:
raise Exception( 'Response was longer than suggested!' )
f.write( next_block )
f.write( block )
for hook in report_hooks: hook( content_length, size_of_response )
next_block = response.read( self.read_block_size )
return ( temp_path, size_of_response )

8
include/HydrusServerResources.py
View File

@ -203,13 +203,7 @@ class HydrusResourceCommand( Resource ):
with open( temp_path, 'wb' ) as f:
block_size = 65536
while True:
block = request.content.read( block_size )
if block == '': break
for block in HC.ReadFileLikeAsBlocks( request.content, 65536 ):
f.write( block )

34
include/HydrusTagArchive.py
View File

@ -61,6 +61,7 @@ class HydrusTagArchive( object ):
if create_db: self._InitDB()
self._namespaces = { namespace for ( namespace, ) in self._c.execute( 'SELECT namespace FROM namespaces;' ) }
self._namespaces.add( '' )
def _InitDB( self ):
@ -157,6 +158,7 @@ class HydrusTagArchive( object ):
def DeleteNamespaces( self ):
self._namespaces = {}
self._namespaces.add( '' )
self._c.execute( 'DELETE FROM namespaces;' )
@ -164,7 +166,22 @@ class HydrusTagArchive( object ):
def GetHashType( self ):
try: ( hash_type, ) = self._c.execute( 'SELECT hash_type FROM hash_type;' ).fetchone()
except: raise Exception( 'This archive has no hash type set.' )
except:
try:
( hash, ) = self._c.execute( 'SELECT hash FROM hashes;' ).fetchone()
if len( hash ) == 16: self.SetHashType( HASH_TYPE_MD5 )
elif len( hash ) == 20: self.SetHashType( HASH_TYPE_SHA1 )
elif len( hash ) == 32: self.SetHashType( HASH_TYPE_SHA256 )
elif len( hash ) == 64: self.SetHashType( HASH_TYPE_SHA512 )
else: raise Exception()
return self.GetHashType()
except: raise Exception( 'This archive has no hash type set, and as it has no files, no hash type guess can be made.' )
return hash_type
@ -192,9 +209,24 @@ class HydrusTagArchive( object ):
except: return False
def IterateMappings( self ):
hash_ids = [ hash_id for ( hash_id, ) in self._c.execute( 'SELECT hash_id FROM hashes;' ) ]
for hash_id in hash_ids:
( hash, ) = self._c.execute( 'SELECT hash FROM hashes WHERE hash_id = ?;', ( hash_id, ) ).fetchone()
tags = self.GetMappings( hash )
if len( tags ) > 0: yield ( hash, tags )
def RebuildNamespaces( self, namespaces_to_exclude = set() ):
self._namespaces = set()
self._namespaces.add( '' )
self._c.execute( 'DELETE FROM namespaces;' )

26
include/HydrusTags.py
View File

@ -153,6 +153,32 @@ def CombineTagSiblingPairs( service_keys_to_statuses_to_pairs ):
return processed_siblings
def FilterNamespaces( tags, namespaces ):
processed_tags = collections.defaultdict( set )
for tag in tags:
if ':' in tag:
( namespace, subtag ) = tag.split( ':', 1 )
processed_tags[ namespace ].add( tag )
else: processed_tags[ '' ].add( tag )
result = set()
for namespace in namespaces:
if namespace in ( '', None ): result.update( processed_tags[ '' ] )
result.update( processed_tags[ namespace ] )
return result
def LoopInSimpleChildrenToParents( simple_children_to_parents, child, parent ):
potential_loop_paths = { parent }

18
include/ServerController.py
View File

@ -160,7 +160,23 @@ class Controller( wx.App ):
self.StartDaemons()
self._tbicon = TaskBarIcon()
if HC.PLATFORM_WINDOWS: self._tbicon = TaskBarIcon()
else:
stay_open_frame = wx.Frame( None, title = 'Hydrus Server' )
stay_open_frame.SetBackgroundColour( wx.SystemSettings.GetColour( wx.SYS_COLOUR_BTNFACE ) )
stay_open_frame.SetIcon( wx.Icon( HC.STATIC_DIR + os.path.sep + 'hydrus.ico', wx.BITMAP_TYPE_ICO ) )
wx.StaticText( stay_open_frame, label = 'The hydrus server is now running.' + os.linesep * 2 + 'Close this window to stop it.' )
( x, y ) = stay_open_frame.GetEffectiveMinSize()
stay_open_frame.SetInitialSize( ( x, y ) )
stay_open_frame.Show()
return True

31
include/ServerDB.py
View File

@ -2195,37 +2195,6 @@ class DB( ServiceDB ):
def _UpdateDB( self, c, version ):
if version == 86:
c.execute( 'COMMIT' )
c.execute( 'PRAGMA foreign_keys = OFF;' )
c.execute( 'BEGIN IMMEDIATE' )
old_service_info = c.execute( 'SELECT * FROM services;' ).fetchall()
c.execute( 'DROP TABLE services;' )
c.execute( 'CREATE TABLE services ( service_id INTEGER PRIMARY KEY, service_key BLOB_BYTES, type INTEGER, options TEXT_YAML );' ).fetchall()
for ( service_id, service_type, port, options ) in old_service_info:
if service_type == HC.SERVER_ADMIN: service_key = 'server admin'
else: service_key = os.urandom( 32 )
options[ 'port' ] = port
c.execute( 'INSERT INTO services ( service_id, service_key, type, options ) VALUES ( ?, ?, ?, ? );', ( service_id, sqlite3.Binary( service_key ), service_type, options ) )
c.execute( 'COMMIT' )
c.execute( 'PRAGMA foreign_keys = ON;' )
c.execute( 'BEGIN IMMEDIATE' )
if version == 90:
for ( service_id, options ) in c.execute( 'SELECT service_id, options FROM services;' ).fetchall():

9
include/TestDialogs.py
View File

@ -162,8 +162,8 @@ class TestNonDBDialogs( unittest.TestCase ):
with ClientGUIDialogs.DialogSelectFromListOfStrings( None, 'select from a list of strings', [ 'a', 'b', 'c' ] ) as dlg:
wx.CallAfter( dlg._strings.Select, 0 )
PressKey( dlg._strings, wx.WXK_SPACE )
wx.CallLater( 500, dlg._strings.Select, 0 )
wx.CallLater( 1000, PressKey, dlg._strings, wx.WXK_SPACE )
result = dlg.ShowModal()
@ -185,9 +185,8 @@ class TestNonDBDialogs( unittest.TestCase ):
with ClientGUIDialogs.DialogSelectFromListOfStrings( None, 'select from a list of strings', [ 'a', 'b', 'c' ] ) as dlg:
wx.CallAfter( dlg._strings.Select, 1 )
HitButton( dlg._ok )
wx.CallLater( 500, dlg._strings.Select, 1 )
wx.CallLater( 1000, HitButton, dlg._ok )
result = dlg.ShowModal()

9
include/TestHydrusServer.py
View File

@ -613,7 +613,14 @@ class TestAMP( unittest.TestCase ):
deferred.addErrback( err )
while not deferred.called: time.sleep( 0.1 )
before = time.time()
while not deferred.called:
time.sleep( 0.1 )
if time.time() - before > 10: raise Exception( 'Trying to get deferred timed out!' )
result = deferred.result

1
server.pyw
View File

@ -49,6 +49,7 @@ try:
print( traceback.format_exc() )
sys.stdout = initial_sys_stdout
sys.stderr = initial_sys_stderr