Version 70

This commit is contained in:
Hydrus 2013-05-15 13:58:14 -05:00
parent 0ff02d3903
commit f8117fe392
25 changed files with 1935 additions and 654 deletions

5
Readme.txt Normal file
View File

@ -0,0 +1,5 @@
If you are reading this, you probably extracted rather than installed, so you were not prompted to read the help.
The hydrus client can do a lot, so do please go into the help folder and open up index.html so you don't get lost. Thanks!
I use a number of the Silk Icons by Mark James at famfamfam.com.

File diff suppressed because it is too large Load Diff

Before

Width:  |  Height:  |  Size: 128 KiB

After

Width:  |  Height:  |  Size: 141 KiB

View File

@ -18,6 +18,7 @@
<li>public tag repository: 4a285629721ca442541ef2c15ea17d1f7f7578b0c3f4f5f2a05f8f0ab297786f@hydrus.no-ip.org:45871</li>
<li>read-only file repository: 8f8a3685abc19e78a92ba61d84a0482b1cfac176fd853f46d93fe437a95e40a5@hydrus.no-ip.org:45872</li>
</ul>
<p><b>Tags are rich, cpu-intensive metadata, so it will take a few minutes for the repository to synchronise completely.</b> The client will lag a little as it processes each update. You can watch its progress in the services->review services dialog as usual.</p>
</div>
</body>
</html>

View File

@ -6,27 +6,31 @@
</head>
<body>
<div class="content">
<p class="warning">This is a draft document. None of this code actually exists yet!</p>
<p class="warning">This is another area under active development right now. I will expand this as I add remote parents.</p>
<h3>what's the problem?</h3>
<p>Tags often fall into certain heirarchies. Files with one particular tag often always have certain other tags, and it is annoying and time-consuming to add them all individually every time.</p>
<p>Tags often fall into certain heirarchies. Files with one certain tags <i>always</i> have certain other tags, and it is annoying and time-consuming to add them all individually every time.</p>
<p>For example, whenever you tag a file with <i>ak-47</i>, you probably also want to tag it <i>assault rifle</i>, and maybe even <i>firearm</i> as well.</p>
<p><img src="tag_parents_venn.png" /></p>
<p>Another time, you might tag a file <i>character:eddard stark</i>, and then also have to type in <i>house stark</i> and then <i>series:game of thrones</i>. (you might also think <i>series:game of thrones</i> should actually be <i>series:a song of ice and fire</i>, but that is an issue for <a href="advanced_siblings.html">siblings</a>)</p>
<p>Drawing more relationships would make a significantly more complicated venn diagram, so let's draw it like a family tree instead:</p>
<p>Drawing more relationships would make a significantly more complicated venn diagram, so let's draw a family tree instead:</p>
<p><img src="tag_parents_got.png" /></p>
<p>If we were to search for the tag <i>house stark</i>, we would expect our results to include all the images of any of the Starks. Even so, it is tedious to have to remember to tag every image of a character with their house affiliation or other parent tags. You may not even remember.</p>
<p>Wouldn't it be great if we could add these sorts of tags at the same time?</p>
<h3>tag parents</h3>
<p>Let's define the relationship 'tag P is tag C's parent' as saying that tag P is the semantic superset of tag C. All files that have C should have P. Let's say that when the user tries to add a tag to a file, the tag's parents are added recursively.</p>
<p>Let's define the relationship 'tag P is tag C's parent' as saying that tag P is the semantic superset/superclass of tag C. <b>All files that have C should have P.</b> Let's say that when the user tries to add a tag to a file, the tag's parents are added recursively.</p>
<p>Let's expand our weapon example:</p>
<p><img src="tag_parents_firearms.png" /></p>
<p>In that graph, adding <i>ar-15</i> to a file would also add <i>semi-automatic rifle</i>, <i>rifle</i>, and <i>firearm</i>. Searching for <i>handgun</i> would everything with <i>m1911</i> and <i>smith and wesson model 10</i>.</p>
<p>Although characters are an obvious and convenient step, we need to be careful to make sure we only link characters to series if they have unique names, and only if they are <i>inextricably</i> linked with their series. Series with crossovers and spinoffs (like CSI) make this difficult.</p>
<p>In that graph, adding <i>ar-15</i> to a file would also add <i>semi-automatic rifle</i>, <i>rifle</i>, and <i>firearm</i>. Searching for <i>handgun</i> would return everything with <i>m1911</i> and <i>smith and wesson model 10</i>.</p>
<p>Although characters are an obvious and convenient step, we need to be careful to make sure we only link characters to series if they have unique names, and only if they are <i>inextricably</i> linked with their series. Series with crossovers and spinoffs (like CSI) make this difficult, so measure twice before you cut.</p>
<h3>how you do it</h3>
<p>services->manage tag parents</p>
<p>picture of dialog</p>
<p>when you add a tag, all of its parents will be added. This is transitive, so grandparents and great-grandparents and so on are also added.</p>
<p>parents are not cast-iron. from time to time, even the simplest and most seemingly-obvious of relationships can be broken. Rule 63 will often break a 'character x is a male' relationship, for example. so, you can de-pend and petition parents without the children being similarly removed. (screenshot?)</p>
<p>Go to <i>services->manage tag parents</i>:</p>
<p><img src="tag_parents_dialog.png" /></p>
<p>Which looks and works just like the manage tag siblings dialog.</p>
<p>Once you have some relationships added, the parents and grandparents will show indented anywhere you 'write' tags, such as the manage tags dialog:</p>
<p><img src="tag_parents_ac_1.png" /></p>
<p>Hitting enter on cersei will try to add <i>house lannister</i> and <i>series:game of thrones</i> as well.</p>
<p><img src="tag_parents_ac_2.png" /></p>
<p><b>Tag parents are not strict!</b> If a parent relationship breaks down for whatever reason (e.g. Rule 63 screwing up character->gender), you can de-pend or petition any parent tag, and the client will not try to remove the children. Tag parents are only ever suggestions in write tag autocomplete dropdowns.</p>
</div>
</body>
</html>

View File

@ -43,7 +43,7 @@
<p><img src="tag_siblings_dialog.png" /></p>
<p>The client will automatically collapse the tagspace to whatever you set. It'll even work with autocomplete, so when you start to type in 'rei ayanami' in search, it will show <i>character:ayanami rei</i> instead:</p>
<p><img src="tag_siblings_rei.png" /></p>
<p>It will not collapse in the write dialog. You will be able to add or remove A as normal, but it will be written in some form of "A (B)" to let you know that, ultimately, the tag will end up displaying in the main gui as B:</p>
<p>It will not collapse in anywhere you 'write' tags, such as the manage tags dialog. You will be able to add or remove A as normal, but it will be written in some form of "A (B)" to let you know that, ultimately, the tag will end up displaying in the main gui as B:</p>
<p><img src="tag_siblings_ac_write.png" /></p>
<p>Although the client may present A as B, it will secretly remember A! You can remove the association A->B, and everything will return to how it was. <b>No information is lost at any point.</b></p>
<p>At the moment, this only supports local siblings. Just like tags, don't go crazy and try to do everything yourself. Have a play with local, but please wait for remote support so your efforts are shared.</p>

View File

@ -8,6 +8,33 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 70</h3></li>
<ul>
<li>tag parents db stuff</li>
<li>tag parents manager</li>
<li>tag parents predicate and matches indented expansion</li>
<li><blockquote>and drawing it!</blockquote></li>
<li>tag parents top result reordering</li>
<li>tag parents top result insertion</li>
<li>tag parents actually doing what they do</li>
<li>polished off tag parent help</li>
<li>made a full-search table to speed up tag A/C requests</li>
<li>added tag full-text search</li>
<li>reworked how tag and file services add and reset, to reduce A/C time</li>
<li>reworked recalc active mappings to be more beautiful, if not faster</li>
<li>fade animation timer improved in several ways</li>
<li>thumbnail fetch made much smoother</li>
<li>newgrounds artist downloading for games and movies added</li>
<li>newgrounds subs added</li>
<li>download panel input now highlights on init</li>
<li>4chan filename tag added</li>
<li>the setfocus on filter close is neater</li>
<li>age phrase is now 'imported [time] ago'</li>
<li>age is now shown in fullscreen</li>
<li>can now copy file from fullscreen, from menu or shortcut</li>
<li>4chan pass authentication improved</li>
<li>fixed a couple tag service precedence sync bugs</li>
</ul>
<li><h3>version 69</h3></li>
<ul>
<li>first dialog for tag siblings</li>
@ -27,7 +54,7 @@
<li>e621 fixed for real this time</li>
<li>A/C improvement that slows tag updates a little but should stop A/C lag after an update</li>
<li>as a result, trying out dropping the CPU-intensive fatten_ac_cache maintenance call</li>
<li>A/C read will now update system preds every time you click on it, so inbex/archive counts will stay accurate</li>
<li>A/C read will now update system preds every time you click on it, so inbox/archive counts will stay accurate</li>
<li>A/C "all known files + tags" will no longer show the mega-laggy total file count</li>
<li>a list -> tuple convenience fix in sanelistctrl</li>
<li>if you don't have any pixiv credentials set up, you will now no longer get the option to start downloading pixiv stuff</li>

View File

@ -14,7 +14,7 @@
<p>In order to share tags with others, you must connect to at least one tag repository. You can create your own, if you like, and share access with whoever you like. I run a public tag repository, which you are very welcome to access and contribute to:</p>
<p><img src="edit_repos_public_tag_repo.png" /></p>
<ul><li>4a285629721ca442541ef2c15ea17d1f7f7578b0c3f4f5f2a05f8f0ab297786f@hydrus.no-ip.org:45871</li></ul>
<p>Tags are rich, cpu-intensive metadata, and it will take a few minutes for the repository to synchronise completely. The client will lag a little as it processes each update. You can watch its progress in the <i>services->review services</i> dialog as usual.</p>
<p><b>Tags are rich, cpu-intensive metadata, so it will take a few minutes for the repository to synchronise completely.</b> The client will lag a little as it processes each update. You can watch its progress in the <i>services->review services</i> dialog as usual.</p>
<h3>now let's find some files</h3>
<p>Once you are all set up, you start to type in a tag, and the autocomplete dropdown will offer suggestions for you:</p>
<p><a href="hidamari_ac.png"><img src="hidamari_ac.png" width="683" height="384" /></a></p>

View File

@ -26,6 +26,7 @@
<li><a href="tagging_schema.html">thoughts on a public tagging schema</a></li>
<li><a href="advanced.html">advanced usage - general</a></li>
<li><a href="advanced_siblings.html">advanced usage - tag siblings</a></li>
<li><a href="advanced_parents.html">advanced usage - tag parents</a></li>
<li><a href="privacy.html">privacy</a></li>
<li><a href="server.html">setting up your own server</a></li>
<li><a href="running_from_source.html">running a client or server from source</a></li>

View File

@ -1,21 +1,20 @@
<html>
<head>
<title>registration keys</title>
<link href="hydrus.ico" rel="shortcut icon" />
<link href="style.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div class="content">
<p class="warning">I threw this together! I'll change it to an easier-to-use wizard or something!</p>
<h3>making new accounts</h3>
<p>Although I hope you are now familiar with access keys, I have recently added <i>registration keys</i>.</p>
<p>These are generated as server admins create new accounts, and are sent to new users. They work only once: to retrieve a new account's access key. After this, they become useless. As a result, they are safer to email and otherwise distribute than access keys.</p>
<p>They look very similar to access keys; their only difference is they have an 'r' at the front, like so:</p>
<p><i>r606927ce4a91114ee81d169ca57ef53fd07c12e3732b01ebcb59ea03d7d5b46e</i></p>
<p>I am still developing how the gui handles these, but for now, the 'add' button on the <i>add, remove or edit services</i> dialog will now give you a simple new popup:</p>
<p><img src="account_registration.png" /></p>
<p>If you have a registration key, click above, and it'll give you a very simple form to fill in. If you have a normal access key, click below, and you'll be able to add the information manually as usual.</p>
<p>For instance, if you want to connect to my public tag repository as per <a href="access_keys.html">here</a>, this is an existing, shared account that has already been initialised, so you would click below.</p>
</div>
</body>
<html>
<head>
<title>registration keys</title>
<link href="hydrus.ico" rel="shortcut icon" />
<link href="style.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div class="content">
<h3>making new accounts</h3>
<p>Although I hope you are now familiar with access keys, I have recently added <i>registration keys</i>.</p>
<p>These are generated as server admins create new accounts, and are sent to new users. They work only once: to retrieve a new account's access key. After this, they become useless. As a result, they are safer to email and otherwise distribute than access keys.</p>
<p>They look very similar to access keys; their only difference is they have an 'r' at the front, like so:</p>
<p><i>r606927ce4a91114ee81d169ca57ef53fd07c12e3732b01ebcb59ea03d7d5b46e</i></p>
<p>I am still developing how the gui handles these, but for now, the 'add' button on the <i>add, remove or edit services</i> dialog will now give you a simple new popup:</p>
<p><img src="account_registration.png" /></p>
<p>If you have a registration key, click above, and it'll give you a very simple form to fill in. If you have a normal access key, click below, and you'll be able to add the information manually as usual.</p>
<p>For instance, if you want to connect to my public tag repository as per <a href="access_keys.html">here</a>, this is an existing, shared account that has already been initialised, so you would click below.</p>
</div>
</body>
</html>

BIN
help/tag_parents_ac_1.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.0 KiB

BIN
help/tag_parents_ac_2.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.0 KiB

BIN
help/tag_parents_dialog.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

View File

@ -604,20 +604,39 @@ class AutocompleteMatchesCounted():
class AutocompleteMatchesPredicates():
def __init__( self, predicates, do_siblings_collapse = True ):
def __init__( self, service_identifier, predicates, collapse = True ):
self._service_identifier = service_identifier
self._predicates = predicates
self._collapse = collapse
siblings_manager = wx.GetApp().GetTagSiblingsManager()
# do siblings
if do_siblings_collapse: self._predicates = siblings_manager.CollapsePredicates( self._predicates )
if self._collapse:
siblings_manager = wx.GetApp().GetTagSiblingsManager()
self._predicates = siblings_manager.CollapsePredicates( self._predicates )
def cmp_func( x, y ): return cmp( x.GetCount(), y.GetCount() )
self._predicates.sort( cmp = cmp_func, reverse = True )
def GetMatches( self, search ): return [ predicate for predicate in self._predicates if HC.SearchEntryMatchesPredicate( search, predicate ) ]
def GetMatches( self, search ):
matches = [ predicate for predicate in self._predicates if HC.SearchEntryMatchesPredicate( search, predicate ) ]
if not self._collapse:
parents_manager = wx.GetApp().GetTagParentsManager()
matches = parents_manager.ExpandPredicates( self._service_identifier, matches )
return matches
class Booru( HC.HydrusYAMLBase ):
@ -2625,6 +2644,98 @@ class ServiceRemoteRestrictedDepotMessage( ServiceRemoteRestrictedDepot ):
def Decrypt( self, encrypted_message ): return HydrusMessageHandling.UnpackageDeliveredMessage( encrypted_message, self._private_key )
class TagParentsManager():
def __init__( self ):
self._RefreshParents()
self._lock = threading.Lock()
HC.pubsub.sub( self, 'RefreshParents', 'notify_new_parents' )
def _GetParents( self, service_identifier, tag ):
if service_identifier == HC.NULL_SERVICE_IDENTIFIER: return []
if tag in self._parents[ service_identifier ]: still_to_process = list( self._parents[ service_identifier ][ tag ] )
else: still_to_process = []
parents = []
parents_set = set()
while len( still_to_process ) > 0:
parent = still_to_process.pop()
if parent in parents_set: continue
parents.append( parent )
parents_set.add( parent )
if parent in self._parents[ service_identifier ]:
grandparents = list( self._parents[ service_identifier ][ parent ] )
still_to_process.extend( grandparents )
return parents
def _RefreshParents( self ):
raw_parents = wx.GetApp().Read( 'tag_parents' )
self._parents = collections.defaultdict( dict )
for ( service_identifier, pairs ) in raw_parents.items(): self._parents[ service_identifier ] = HC.BuildKeyToListDict( pairs )
def ExpandPredicates( self, service_identifier, predicates ):
if service_identifier == HC.NULL_SERVICE_IDENTIFIER: return predicates
results = []
with self._lock:
for predicate in predicates:
results.append( predicate )
if predicate.GetPredicateType() == HC.PREDICATE_TYPE_TAG:
tag = predicate.GetTag()
parents = self._GetParents( service_identifier, tag )
for parent in parents:
parent_predicate = HC.Predicate( HC.PREDICATE_TYPE_PARENT, parent, None )
results.append( parent_predicate )
return results
def GetParents( self, service_identifier, tag ):
with self._lock: return self._GetParents( service_identifier, tag )
def RefreshParents( self ):
with self._lock: self._RefreshParents()
class TagSiblingsManager():
def __init__( self ):
@ -2871,28 +2982,62 @@ class ThumbnailCache():
def GetNotFoundThumbnail( self ): return self._not_found
def GetPDFThumbnail( self ): return self._pdf
def GetThumbnail( self, hash ):
def GetThumbnail( self, media ):
if not self._data_cache.HasData( hash ):
try: hydrus_bitmap = HydrusImageHandling.GenerateHydrusBitmapFromFile( wx.GetApp().Read( 'thumbnail', hash ) )
except:
print( traceback.format_exc() )
return self._not_found
self._data_cache.AddData( hash, hydrus_bitmap )
mime = media.GetDisplayMedia().GetMime()
return self._data_cache.GetData( hash )
if mime in HC.IMAGES:
hash = media.GetDisplayMedia().GetHash()
if not self._data_cache.HasData( hash ):
try: hydrus_bitmap = HydrusImageHandling.GenerateHydrusBitmapFromFile( wx.GetApp().Read( 'thumbnail', hash ) )
except:
print( traceback.format_exc() )
return self._not_found
self._data_cache.AddData( hash, hydrus_bitmap )
return self._data_cache.GetData( hash )
elif mime == HC.APPLICATION_FLASH: return self._flash
elif mime == HC.APPLICATION_PDF: return self._pdf
elif mime == HC.VIDEO_FLV: return self._flv
else: return self._not_found
def Prefetch( self, hashes ):
hashes_to_get = hashes.intersection( self._data_cache.GetKeys() )
hashes_to_get = hashes.difference( self._data_cache.GetKeys() )
if len( hashes_to_get ) > 0: threading.Thread( target = self.THREADPrefetch, args = ( hashes_to_get, ) ).start()
def Waterfall( self, page_key, medias ): threading.Thread( target = self.THREADWaterfall, args = ( page_key, medias ) ).start()
def THREADWaterfall( self, page_key, medias ):
random.shuffle( medias )
last_paused = time.clock()
for media in medias:
thumbnail = self.GetThumbnail( media )
HC.pubsub.pub( 'waterfall_thumbnail', page_key, media, thumbnail )
if time.clock() - last_paused > 0.005:
time.sleep( 0.0001 )
last_paused = time.clock()
def THREADPrefetch( self, hashes ):
for hash in hashes: wx.GetApp().ReadDaemon( 'thumbnail', hash )

View File

@ -131,6 +131,8 @@ class Controller( wx.App ):
def GetSessionKey( self, service_identifier ): return self._session_manager.GetSessionKey( service_identifier )
def GetTagParentsManager( self ): return self._tag_parents_manager
def GetTagSiblingsManager( self ): return self._tag_siblings_manager
def GetThumbnailCache( self ): return self._thumbnail_cache
@ -171,6 +173,7 @@ class Controller( wx.App ):
self._session_manager = HydrusSessions.HydrusSessionManagerClient()
self._web_session_manager = CC.WebSessionManagerClient()
self._tag_parents_manager = CC.TagParentsManager()
self._tag_siblings_manager = CC.TagSiblingsManager()
self.SetSplashText( 'caches' )

View File

@ -1233,7 +1233,14 @@ class TagDB():
tags_not_in_db.difference_update( [ tag for ( tag, ) in c.execute( 'SELECT tag FROM tags WHERE tag IN (' + ','.join( '?' * len( tags_subset ) ) + ');', [ tag for tag in tags_subset ] ) ] )
if len( tags_not_in_db ) > 0: c.executemany( 'INSERT INTO tags( tag ) VALUES( ? );', [ ( tag, ) for tag in tags_not_in_db ] )
if len( tags_not_in_db ) > 0:
inserts = [ ( tag, ) for tag in tags_not_in_db ]
c.executemany( 'INSERT INTO tags ( tag ) VALUES ( ? );', inserts )
c.executemany( 'INSERT INTO tags_fts4 ( docid, tag ) SELECT tag_id, tag FROM tags WHERE tag = ?;', inserts )
def _GetNamespaceTag( self, c, namespace_id, tag_id ):
@ -1285,6 +1292,8 @@ class TagDB():
tag_id = c.lastrowid
c.execute( 'INSERT INTO tags_fts4 ( docid, tag ) VALUES ( ?, ? );', ( tag_id, tag ) )
else: ( tag_id, ) = result
result = c.execute( 'SELECT 1 FROM existing_tags WHERE namespace_id = ? AND tag_id = ?;', ( namespace_id, tag_id ) ).fetchone()
@ -1457,7 +1466,40 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
c.execute( 'INSERT OR IGNORE INTO repositories ( service_id, first_begin, next_begin ) VALUES ( ?, ?, ? );', ( service_id, 0, 0 ) )
if service_type == HC.TAG_REPOSITORY: c.execute( 'INSERT INTO tag_service_precedence ( service_id, precedence ) SELECT ?, CASE WHEN MAX( precedence ) NOT NULL THEN MAX( precedence ) + 1 ELSE 0 END FROM tag_service_precedence;', ( service_id, ) )
if service_type == HC.TAG_REPOSITORY:
c.execute( 'INSERT INTO tag_service_precedence ( service_id, precedence ) SELECT ?, CASE WHEN MAX( precedence ) NOT NULL THEN MAX( precedence ) + 1 ELSE 0 END FROM tag_service_precedence;', ( service_id, ) )
self._RebuildTagServicePrecedence( c )
#
file_service_identifiers = self._GetServiceIdentifiers( c, ( HC.FILE_REPOSITORY, HC.LOCAL_FILE ) )
file_service_identifiers.add( HC.NULL_SERVICE_IDENTIFIER )
file_service_ids = [ self._GetServiceId( c, service_identifier ) for service_identifier in file_service_identifiers ]
existing_tag_ids = c.execute( 'SELECT namespace_id, tag_id FROM existing_tags;' ).fetchall()
inserts = ( ( file_service_id, service_id, namespace_id, tag_id, 0, 0 ) for ( file_service_id, ( namespace_id, tag_id ) ) in itertools.product( file_service_ids, existing_tag_ids ) )
c.executemany( 'INSERT OR IGNORE INTO autocomplete_tags_cache ( file_service_id, tag_service_id, namespace_id, tag_id, current_count, pending_count ) VALUES ( ?, ?, ?, ?, ?, ? );', inserts )
elif service_type == HC.FILE_REPOSITORY:
tag_service_identifiers = self._GetServiceIdentifiers( c, ( HC.TAG_REPOSITORY, HC.LOCAL_TAG ) )
tag_service_identifiers.add( HC.NULL_SERVICE_IDENTIFIER )
tag_service_ids = [ self._GetServiceId( c, service_identifier ) for service_identifier in tag_service_identifiers ]
existing_tag_ids = c.execute( 'SELECT namespace_id, tag_id FROM existing_tags;' ).fetchall()
inserts = ( ( service_id, tag_service_id, namespace_id, tag_id, 0, 0 ) for ( tag_service_id, ( namespace_id, tag_id ) ) in itertools.product( tag_service_ids, existing_tag_ids ) )
c.executemany( 'INSERT OR IGNORE INTO autocomplete_tags_cache ( file_service_id, tag_service_id, namespace_id, tag_id, current_count, pending_count ) VALUES ( ?, ?, ?, ?, ?, ? );', inserts )
elif service_type == HC.RATING_LIKE_REPOSITORY:
( like, dislike ) = extra_info
@ -2064,7 +2106,7 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
return all_downloads
def _GetAutocompleteTags( self, c, tag_service_identifier = HC.NULL_SERVICE_IDENTIFIER, file_service_identifier = HC.NULL_SERVICE_IDENTIFIER, half_complete_tag = '', include_current = True, include_pending = True, do_siblings_collapse = True ):
def _GetAutocompleteTags( self, c, tag_service_identifier = HC.NULL_SERVICE_IDENTIFIER, file_service_identifier = HC.NULL_SERVICE_IDENTIFIER, half_complete_tag = '', include_current = True, include_pending = True, collapse = True ):
if tag_service_identifier == HC.NULL_SERVICE_IDENTIFIER:
@ -2133,7 +2175,9 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
( namespace_id, ) = result
possible_tag_ids = [ tag_id for ( tag_id, ) in c.execute( 'SELECT tag_id FROM tags WHERE tag LIKE ?;', ( half_complete_tag + '%', ) ) ]
#possible_tag_ids = [ tag_id for ( tag_id, ) in c.execute( 'SELECT tag_id FROM tags WHERE tag LIKE ?;', ( half_complete_tag + '%', ) ) ]
possible_tag_ids = [ tag_id for ( tag_id, ) in c.execute( 'SELECT docid FROM tags_fts4 WHERE tag MATCH ?;', ( '"' + half_complete_tag + '*"', ) ) ]
predicates_phrase = 'namespace_id = ' + str( namespace_id ) + ' AND tag_id IN ' + HC.SplayListForDB( possible_tag_ids )
@ -2141,7 +2185,9 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
else:
possible_tag_ids = [ tag_id for ( tag_id, ) in c.execute( 'SELECT tag_id FROM tags WHERE tag LIKE ?;', ( half_complete_tag + '%', ) ) ]
#possible_tag_ids = [ tag_id for ( tag_id, ) in c.execute( 'SELECT tag_id FROM tags WHERE tag LIKE ?;', ( half_complete_tag + '%', ) ) ]
possible_tag_ids = [ tag_id for ( tag_id, ) in c.execute( 'SELECT docid FROM tags_fts4 WHERE tag MATCH ?;', ( '"' + half_complete_tag + '*"', ) ) ]
predicates_phrase = 'tag_id IN ' + HC.SplayListForDB( possible_tag_ids )
@ -2211,7 +2257,7 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
[ tags_to_count.update( { ( 1, tag_id ) : num_tags } ) for ( namespace_id, tag_id, num_tags ) in results if namespace_id != 1 and tag_id in unnamespaced_tag_ids ]
matches = CC.AutocompleteMatchesPredicates( [ HC.Predicate( HC.PREDICATE_TYPE_TAG, ( '+', self._GetNamespaceTag( c, namespace_id, tag_id ) ), num_tags ) for ( ( namespace_id, tag_id ), num_tags ) in tags_to_count.items() if num_tags > 0 ], do_siblings_collapse = do_siblings_collapse )
matches = CC.AutocompleteMatchesPredicates( tag_service_identifier, [ HC.Predicate( HC.PREDICATE_TYPE_TAG, ( '+', self._GetNamespaceTag( c, namespace_id, tag_id ) ), num_tags ) for ( ( namespace_id, tag_id ), num_tags ) in tags_to_count.items() if num_tags > 0 ], collapse = collapse )
return matches
@ -3178,6 +3224,37 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
return subscriptions
def _GetTagParents( self, c, service_identifier = None ):
if service_identifier is None:
results = HC.BuildKeyToListDict( ( ( service_id, ( old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) ) for ( service_id, old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) in c.execute( 'SELECT service_id, old_namespace_id, old_tag_id, new_namespace_id, new_tag_id FROM tag_parents;' ) ) )
processed_results = {}
for ( service_id, pair_ids ) in results.items():
service_identifier = self._GetServiceIdentifier( c, service_id )
pairs = [ ( self._GetNamespaceTag( c, old_namespace_id, old_tag_id ), self._GetNamespaceTag( c, new_namespace_id, new_tag_id ) ) for ( old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) in pair_ids ]
processed_results[ service_identifier ] = pairs
return processed_results
else:
service_id = self._GetServiceId( c, service_identifier )
pair_ids = c.execute( 'SELECT old_namespace_id, old_tag_id, new_namespace_id, new_tag_id FROM tag_parents WHERE service_id = ?;', ( service_id, ) ).fetchall()
pairs = [ ( self._GetNamespaceTag( c, old_namespace_id, old_tag_id ), self._GetNamespaceTag( c, new_namespace_id, new_tag_id ) ) for ( old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) in pair_ids ]
return pairs
def _GetTagServicePrecedence( self, c ):
service_ids = [ service_id for ( service_id, ) in c.execute( 'SELECT service_id FROM tag_service_precedence ORDER BY precedence ASC;' ) ]
@ -3195,13 +3272,13 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
processed_results = {}
for ( service_id, n_t_ids ) in results.items():
for ( service_id, pair_ids ) in results.items():
service_identifier = self._GetServiceIdentifier( c, service_id )
n_ts = [ ( self._GetNamespaceTag( c, old_namespace_id, old_tag_id ), self._GetNamespaceTag( c, new_namespace_id, new_tag_id ) ) for ( old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) in n_t_ids ]
pairs = [ ( self._GetNamespaceTag( c, old_namespace_id, old_tag_id ), self._GetNamespaceTag( c, new_namespace_id, new_tag_id ) ) for ( old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) in pair_ids ]
processed_results[ service_identifier ] = n_ts
processed_results[ service_identifier ] = pairs
return processed_results
@ -3210,11 +3287,11 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
service_id = self._GetServiceId( c, service_identifier )
n_t_ids = c.execute( 'SELECT old_namespace_id, old_tag_id, new_namespace_id, new_tag_id FROM tag_siblings WHERE service_id = ?;', ( service_id, ) ).fetchall()
pair_ids = c.execute( 'SELECT old_namespace_id, old_tag_id, new_namespace_id, new_tag_id FROM tag_siblings WHERE service_id = ?;', ( service_id, ) ).fetchall()
n_ts = [ ( self._GetNamespaceTag( c, old_namespace_id, old_tag_id ), self._GetNamespaceTag( c, new_namespace_id, new_tag_id ) ) for ( old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) in n_t_ids ]
pairs = [ ( self._GetNamespaceTag( c, old_namespace_id, old_tag_id ), self._GetNamespaceTag( c, new_namespace_id, new_tag_id ) ) for ( old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) in pair_ids ]
return n_ts
return pairs
@ -3752,6 +3829,8 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
service_ids = [ service_id for ( service_id, ) in c.execute( 'SELECT service_id FROM tag_service_precedence ORDER BY precedence DESC;' ) ]
t = time.clock()
c.execute( 'DELETE FROM active_mappings;' )
c.execute( 'DELETE FROM active_pending_mappings;' )
@ -3762,12 +3841,12 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
c.execute( 'INSERT OR IGNORE INTO active_mappings SELECT namespace_id, tag_id, hash_id FROM mappings WHERE service_id = ?;', ( service_id, ) )
c.execute( 'INSERT OR IGNORE INTO active_pending_mappings SELECT namespace_id, tag_id, hash_id FROM pending_mappings WHERE service_id = ?;', ( service_id, ) )
# is this incredibly inefficient?
# if this is O( n-squared ) or whatever, just rewrite it as two queries using indices
if not first_round:
c.execute( 'DELETE FROM active_mappings WHERE namespace_id || "," || tag_id || "," || hash_id IN ( SELECT namespace_id || "," || tag_id || "," || hash_id FROM deleted_mappings WHERE service_id = ? );', ( service_id, ) )
c.execute( 'DELETE FROM active_pending_mappings WHERE namespace_id || "," || tag_id || "," || hash_id IN ( SELECT namespace_id || "," || tag_id || "," || hash_id FROM deleted_mappings WHERE service_id = ? );', ( service_id, ) )
ids_dict = HC.BuildKeyToListDict( [ ( ( namespace_id, tag_id ), hash_id ) for ( namespace_id, tag_id, hash_id ) in c.execute( 'SELECT namespace_id, tag_id, hash_id FROM deleted_mappings WHERE service_id = ?;', ( service_id, ) ) ] )
c.executemany( 'DELETE FROM active_mappings WHERE namespace_id = ? AND tag_id = ? AND hash_id = ?;', ( ( namespace_id, tag_id, hash_id ) for ( ( namespace_id, tag_id ), hash_id ) in ids_dict.items() ) )
c.executemany( 'DELETE FROM active_pending_mappings WHERE namespace_id = ? AND tag_id = ? AND hash_id = ?;', ( ( namespace_id, tag_id, hash_id ) for ( ( namespace_id, tag_id ), hash_id ) in ids_dict.items() ) )
first_round = False
@ -3775,6 +3854,12 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
c.execute( 'DELETE FROM autocomplete_tags_cache WHERE tag_service_id IS NULL;' )
file_service_identifiers = self._GetServiceIdentifiers( c, ( HC.FILE_REPOSITORY, HC.LOCAL_FILE ) )
file_service_identifiers.add( HC.NULL_SERVICE_IDENTIFIER )
for file_service_identifier in file_service_identifiers: self._GetAutocompleteTags( c, file_service_identifier = file_service_identifier )
def _RecalcActivePendingMappings( self, c ):
@ -3788,9 +3873,12 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
c.execute( 'INSERT OR IGNORE INTO active_pending_mappings SELECT namespace_id, tag_id, hash_id FROM pending_mappings WHERE service_id = ?;', ( service_id, ) )
# is this incredibly inefficient?
# if this is O( n-squared ) or whatever, just rewrite it as two queries using indices
if not first_round: c.execute( 'DELETE FROM active_pending_mappings WHERE namespace_id || "," || tag_id || "," || hash_id IN ( SELECT namespace_id || "," || tag_id || "," || hash_id FROM deleted_mappings WHERE service_id = ? );', ( service_id, ) )
if not first_round:
ids_dict = HC.BuildKeyToListDict( [ ( ( namespace_id, tag_id ), hash_id ) for ( namespace_id, tag_id, hash_id ) in c.execute( 'SELECT namespace_id, tag_id, hash_id FROM deleted_mappings WHERE service_id = ?;', ( service_id, ) ) ] )
c.executemany( 'DELETE FROM active_pending_mappings WHERE namespace_id = ? AND tag_id = ? AND hash_id = ?;', ( ( namespace_id, tag_id, hash_id ) for ( ( namespace_id, tag_id ), hash_id ) in ids_dict.items() ) )
first_round = False
@ -3826,7 +3914,12 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
c.execute( 'DELETE FROM services WHERE service_id = ?;', ( service_id, ) )
if service_type == HC.TAG_REPOSITORY: self._RecalcActiveMappings( c )
if service_type == HC.TAG_REPOSITORY:
self._RebuildTagServicePrecedence( c )
self._RecalcActiveMappings( c )
self._AddService( c, new_service_identifier, credentials, extra_info )
@ -3881,18 +3974,25 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
self.pub( 'notify_new_subscriptions' )
def _SetTagServicePrecedence( self, c, service_identifiers ):
def _RebuildTagServicePrecedence( self, c ):
del self._tag_service_precedence[:]
service_identifiers = self._GetTagServicePrecedence( c )
self._tag_service_precedence.extend( service_identifiers )
service_ids = [ self._GetServiceId( c, service_identifier ) for service_identifier in service_identifiers ]
def _SetTagServicePrecedence( self, c, service_identifiers ):
c.execute( 'DELETE FROM tag_service_precedence;' )
service_ids = [ self._GetServiceId( c, service_identifier ) for service_identifier in service_identifiers ]
c.executemany( 'INSERT INTO tag_service_precedence ( service_id, precedence ) VALUES ( ?, ? );', [ ( service_id, precedence ) for ( precedence, service_id ) in enumerate( service_ids ) ] )
self._RebuildTagServicePrecedence( c )
self._RecalcActiveMappings( c )
@ -4180,7 +4280,12 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
if recalc_active_mappings: self._RecalcActiveMappings( c )
if recalc_active_mappings:
self._RebuildTagServicePrecedence( c )
self._RecalcActiveMappings( c )
self.pub( 'notify_new_pending' )
self.pub( 'notify_new_services' )
@ -4279,7 +4384,12 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
if recalc_active_mappings: self._RecalcActiveMappings( c )
if recalc_active_mappings:
self._RebuildTagServicePrecedence( c )
self._RecalcActiveMappings( c )
self.pub( 'notify_new_pending' )
self.pub( 'notify_new_services' )
@ -4495,6 +4605,54 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
def _UpdateTagParents( self, c, edit_log ):
pending_updated = False
for ( service_identifier, sub_edit_log ) in edit_log:
service_id = self._GetServiceId( c, service_identifier )
for ( action, data ) in sub_edit_log:
if action == HC.ADD:
( old_tag, new_tag ) = data
( old_namespace_id, old_tag_id ) = self._GetNamespaceIdTagId( c, old_tag )
( new_namespace_id, new_tag_id ) = self._GetNamespaceIdTagId( c, new_tag )
c.execute( 'INSERT OR IGNORE INTO tag_parents ( service_id, old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) VALUES ( ?, ?, ?, ?, ? );', ( service_id, old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) )
#
existing_hash_ids = [ hash for ( hash, ) in c.execute( 'SELECT hash_id FROM mappings WHERE service_id = ? AND namespace_id = ? AND tag_id = ?;', ( service_id, old_namespace_id, new_tag_id ) ) ]
new_mapping_ids = [ ( new_namespace_id, new_tag_id, existing_hash_ids ) ]
self._UpdateMappings( c, service_id, new_mapping_ids, [] )
# also catch up pending, for remote
# this is a process content update thing, right?
# I should extract it to a new self._UpdatePendingMappings function
elif action == HC.DELETE:
( old_tag, new_tag ) = data
( old_namespace_id, old_tag_id ) = self._GetNamespaceIdTagId( c, old_tag )
( new_namespace_id, new_tag_id ) = self._GetNamespaceIdTagId( c, new_tag )
c.execute( 'DELETE FROM tag_parents WHERE service_id = ? AND old_namespace_id = ? AND old_tag_id = ? AND new_namespace_id = ? AND new_tag_id = ?;', ( service_id, old_namespace_id, old_tag_id, new_namespace_id, new_tag_id ) )
self.pub( 'notify_new_parents' )
def _UpdateTagSiblings( self, c, edit_log ):
for ( service_identifier, sub_edit_log ) in edit_log:
@ -4571,6 +4729,21 @@ class DB( ServiceDB ):
# ####### put a temp db update here! ######
c.execute( 'BEGIN IMMEDIATE' )
try:
c.execute( 'COMMIT' )
except:
print( traceback.format_exc() )
c.execute( 'ROLLBACK' )
raise
# ###### ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ######
( self._local_file_service_id, ) = c.execute( 'SELECT service_id FROM services WHERE type = ?;', ( HC.LOCAL_FILE, ) ).fetchone()
@ -4877,11 +5050,16 @@ class DB( ServiceDB ):
c.execute( 'CREATE TABLE tag_service_precedence ( service_id INTEGER PRIMARY KEY REFERENCES services ON DELETE CASCADE, precedence INTEGER );' )
c.execute( 'CREATE TABLE tag_parents ( service_id INTEGER REFERENCES services ON DELETE CASCADE, old_namespace_id INTEGER, old_tag_id INTEGER, new_namespace_id INTEGER, new_tag_id INTEGER );' )
c.execute( 'CREATE UNIQUE INDEX tag_parents_all_index ON tag_parents ( service_id, old_namespace_id, old_tag_id, new_namespace_id, new_tag_id );' )
c.execute( 'CREATE TABLE tag_siblings ( service_id INTEGER REFERENCES services ON DELETE CASCADE, old_namespace_id INTEGER, old_tag_id INTEGER, new_namespace_id INTEGER, new_tag_id INTEGER, PRIMARY KEY ( service_id, old_namespace_id, old_tag_id ) );' )
c.execute( 'CREATE TABLE tags ( tag_id INTEGER PRIMARY KEY, tag TEXT );' )
c.execute( 'CREATE UNIQUE INDEX tags_tag_index ON tags ( tag );' )
c.execute( 'CREATE VIRTUAL TABLE tags_fts4 USING fts4( tag );' )
c.execute( 'CREATE TABLE urls ( url TEXT PRIMARY KEY, hash_id INTEGER );' )
c.execute( 'CREATE INDEX urls_hash_id ON urls ( hash_id );' )
@ -5166,158 +5344,6 @@ class DB( ServiceDB ):
self._UpdateDBOld( c, version )
if version < 64:
c.execute( 'CREATE TABLE web_sessions ( name TEXT PRIMARY KEY, cookies TEXT_YAML, expiry INTEGER );' )
c.execute( 'UPDATE ADDRESSES SET host = ? WHERE host = ?;', ( 'hydrus.no-ip.org', '98.214.1.156' ) )
c.execute( 'DELETE FROM service_info WHERE info_type IN ( 6, 7 );' ) # resetting thumb count, to see if it breaks again
( self._options, ) = c.execute( 'SELECT options FROM options;' ).fetchone()
shortcuts = self._options[ 'shortcuts' ]
shortcuts[ wx.ACCEL_SHIFT ][ wx.WXK_UP ] = 'pan_up'
shortcuts[ wx.ACCEL_SHIFT ][ wx.WXK_DOWN ] = 'pan_down'
shortcuts[ wx.ACCEL_SHIFT ][ wx.WXK_LEFT ] = 'pan_left'
shortcuts[ wx.ACCEL_SHIFT ][ wx.WXK_RIGHT ] = 'pan_right'
c.execute( 'UPDATE options SET options = ?;', ( self._options, ) )
if version < 65:
wx.GetApp().SetSplashText( 'renaming db files' )
filenames = dircache.listdir( HC.CLIENT_FILES_DIR )
i = 1
for filename in filenames:
if '.' not in filename:
try:
old_path = HC.CLIENT_FILES_DIR + os.path.sep + filename
mime = HC.GetMimeFromPath( old_path )
new_path = old_path + HC.mime_ext_lookup[ mime ]
shutil.move( old_path, new_path )
os.chmod( new_path, stat.S_IREAD )
except: pass
i += 1
if i % 250 == 0: wx.GetApp().SetSplashText( 'renaming file ' + HC.ConvertIntToPrettyString( i ) + '/' + HC.ConvertIntToPrettyString( len( filenames ) ) )
c.execute( 'CREATE TABLE subscriptions ( subscriptions TEXT_YAML );' )
if version < 66:
c.execute( 'DELETE FROM boorus;' )
c.executemany( 'INSERT INTO boorus VALUES ( ?, ? );', [ ( booru.GetName(), booru ) for booru in CC.DEFAULT_BOORUS ] )
( self._options, ) = c.execute( 'SELECT options FROM options;' ).fetchone()
self._options[ 'pause_repo_sync' ] = False
self._options[ 'pause_subs_sync' ] = False
c.execute( 'UPDATE options SET options = ?;', ( self._options, ) )
if version < 67:
result = c.execute( 'SELECT subscriptions FROM subscriptions;' ).fetchone()
if result is None: subscriptions = []
else: ( subscriptions, ) = result
c.execute( 'DROP TABLE subscriptions;' )
c.execute( 'CREATE TABLE subscriptions ( site_download_type INTEGER, name TEXT, info TEXT_YAML, PRIMARY KEY( site_download_type, name ) );' )
inserts = [ ( site_download_type, name, [ query_type, query, frequency_type, frequency_number, advanced_tag_options, advanced_import_options, last_checked, url_cache ] ) for ( site_download_type, name, query_type, query, frequency_type, frequency_number, advanced_tag_options, advanced_import_options, last_checked, url_cache ) in subscriptions ]
c.executemany( 'INSERT INTO subscriptions ( site_download_type, name, info ) VALUES ( ?, ?, ? );', inserts )
#
wx.GetApp().SetSplashText( 'creating new db directories' )
hex_chars = '0123456789abcdef'
for ( one, two ) in itertools.product( hex_chars, hex_chars ):
dir = HC.CLIENT_FILES_DIR + os.path.sep + one + two
if not os.path.exists( dir ): os.mkdir( dir )
dir = HC.CLIENT_THUMBNAILS_DIR + os.path.sep + one + two
if not os.path.exists( dir ): os.mkdir( dir )
wx.GetApp().SetSplashText( 'generating file cache' )
filenames = dircache.listdir( HC.CLIENT_FILES_DIR )
i = 1
for filename in filenames:
try:
source_path = HC.CLIENT_FILES_DIR + os.path.sep + filename
first_two_chars = filename[:2]
destination_path = HC.CLIENT_FILES_DIR + os.path.sep + first_two_chars + os.path.sep + filename
shutil.move( source_path, destination_path )
except: continue
i += 1
if i % 100 == 0: wx.GetApp().SetSplashText( 'moving files - ' + HC.ConvertIntToPrettyString( i ) + '/' + HC.ConvertIntToPrettyString( len( filenames ) ) )
wx.GetApp().SetSplashText( 'generating thumbnail cache' )
filenames = dircache.listdir( HC.CLIENT_THUMBNAILS_DIR )
i = 1
for filename in filenames:
try:
source_path = HC.CLIENT_THUMBNAILS_DIR + os.path.sep + filename
first_two_chars = filename[:2]
destination_path = HC.CLIENT_THUMBNAILS_DIR + os.path.sep + first_two_chars + os.path.sep + filename
shutil.move( source_path, destination_path )
except: continue
i += 1
if i % 100 == 0: wx.GetApp().SetSplashText( 'moving thumbnails - ' + HC.ConvertIntToPrettyString( i ) + '/' + HC.ConvertIntToPrettyString( len( filenames ) ) )
if version < 68:
( self._options, ) = c.execute( 'SELECT options FROM options;' ).fetchone()
@ -5420,6 +5446,17 @@ class DB( ServiceDB ):
except: pass
if version < 70:
c.execute( 'CREATE TABLE tag_parents ( service_id INTEGER REFERENCES services ON DELETE CASCADE, old_namespace_id INTEGER, old_tag_id INTEGER, new_namespace_id INTEGER, new_tag_id INTEGER );' )
c.execute( 'CREATE UNIQUE INDEX tag_parents_all_index ON tag_parents ( service_id, old_namespace_id, old_tag_id, new_namespace_id, new_tag_id );' )
#
c.execute( 'CREATE VIRTUAL TABLE tags_fts4 USING fts4( tag );' )
c.execute( 'INSERT INTO tags_fts4 ( docid, tag ) SELECT tag_id, tag FROM tags;' )
unknown_account = CC.GetUnknownAccount()
unknown_account.MakeStale()
@ -6420,6 +6457,158 @@ class DB( ServiceDB ):
c.execute( 'UPDATE options SET options = ?;', ( self._options, ) )
if version < 64:
c.execute( 'CREATE TABLE web_sessions ( name TEXT PRIMARY KEY, cookies TEXT_YAML, expiry INTEGER );' )
c.execute( 'UPDATE ADDRESSES SET host = ? WHERE host = ?;', ( 'hydrus.no-ip.org', '98.214.1.156' ) )
c.execute( 'DELETE FROM service_info WHERE info_type IN ( 6, 7 );' ) # resetting thumb count, to see if it breaks again
( self._options, ) = c.execute( 'SELECT options FROM options;' ).fetchone()
shortcuts = self._options[ 'shortcuts' ]
shortcuts[ wx.ACCEL_SHIFT ][ wx.WXK_UP ] = 'pan_up'
shortcuts[ wx.ACCEL_SHIFT ][ wx.WXK_DOWN ] = 'pan_down'
shortcuts[ wx.ACCEL_SHIFT ][ wx.WXK_LEFT ] = 'pan_left'
shortcuts[ wx.ACCEL_SHIFT ][ wx.WXK_RIGHT ] = 'pan_right'
c.execute( 'UPDATE options SET options = ?;', ( self._options, ) )
if version < 65:
wx.GetApp().SetSplashText( 'renaming db files' )
filenames = dircache.listdir( HC.CLIENT_FILES_DIR )
i = 1
for filename in filenames:
if '.' not in filename:
try:
old_path = HC.CLIENT_FILES_DIR + os.path.sep + filename
mime = HC.GetMimeFromPath( old_path )
new_path = old_path + HC.mime_ext_lookup[ mime ]
shutil.move( old_path, new_path )
os.chmod( new_path, stat.S_IREAD )
except: pass
i += 1
if i % 250 == 0: wx.GetApp().SetSplashText( 'renaming file ' + HC.ConvertIntToPrettyString( i ) + '/' + HC.ConvertIntToPrettyString( len( filenames ) ) )
c.execute( 'CREATE TABLE subscriptions ( subscriptions TEXT_YAML );' )
if version < 66:
c.execute( 'DELETE FROM boorus;' )
c.executemany( 'INSERT INTO boorus VALUES ( ?, ? );', [ ( booru.GetName(), booru ) for booru in CC.DEFAULT_BOORUS ] )
( self._options, ) = c.execute( 'SELECT options FROM options;' ).fetchone()
self._options[ 'pause_repo_sync' ] = False
self._options[ 'pause_subs_sync' ] = False
c.execute( 'UPDATE options SET options = ?;', ( self._options, ) )
if version < 67:
result = c.execute( 'SELECT subscriptions FROM subscriptions;' ).fetchone()
if result is None: subscriptions = []
else: ( subscriptions, ) = result
c.execute( 'DROP TABLE subscriptions;' )
c.execute( 'CREATE TABLE subscriptions ( site_download_type INTEGER, name TEXT, info TEXT_YAML, PRIMARY KEY( site_download_type, name ) );' )
inserts = [ ( site_download_type, name, [ query_type, query, frequency_type, frequency_number, advanced_tag_options, advanced_import_options, last_checked, url_cache ] ) for ( site_download_type, name, query_type, query, frequency_type, frequency_number, advanced_tag_options, advanced_import_options, last_checked, url_cache ) in subscriptions ]
c.executemany( 'INSERT INTO subscriptions ( site_download_type, name, info ) VALUES ( ?, ?, ? );', inserts )
#
wx.GetApp().SetSplashText( 'creating new db directories' )
hex_chars = '0123456789abcdef'
for ( one, two ) in itertools.product( hex_chars, hex_chars ):
dir = HC.CLIENT_FILES_DIR + os.path.sep + one + two
if not os.path.exists( dir ): os.mkdir( dir )
dir = HC.CLIENT_THUMBNAILS_DIR + os.path.sep + one + two
if not os.path.exists( dir ): os.mkdir( dir )
wx.GetApp().SetSplashText( 'generating file cache' )
filenames = dircache.listdir( HC.CLIENT_FILES_DIR )
i = 1
for filename in filenames:
try:
source_path = HC.CLIENT_FILES_DIR + os.path.sep + filename
first_two_chars = filename[:2]
destination_path = HC.CLIENT_FILES_DIR + os.path.sep + first_two_chars + os.path.sep + filename
shutil.move( source_path, destination_path )
except: continue
i += 1
if i % 100 == 0: wx.GetApp().SetSplashText( 'moving files - ' + HC.ConvertIntToPrettyString( i ) + '/' + HC.ConvertIntToPrettyString( len( filenames ) ) )
wx.GetApp().SetSplashText( 'generating thumbnail cache' )
filenames = dircache.listdir( HC.CLIENT_THUMBNAILS_DIR )
i = 1
for filename in filenames:
try:
source_path = HC.CLIENT_THUMBNAILS_DIR + os.path.sep + filename
first_two_chars = filename[:2]
destination_path = HC.CLIENT_THUMBNAILS_DIR + os.path.sep + first_two_chars + os.path.sep + filename
shutil.move( source_path, destination_path )
except: continue
i += 1
if i % 100 == 0: wx.GetApp().SetSplashText( 'moving thumbnails - ' + HC.ConvertIntToPrettyString( i ) + '/' + HC.ConvertIntToPrettyString( len( filenames ) ) )
def _UpdateDBOldPost( self, c, version ):
@ -7440,6 +7629,7 @@ class DB( ServiceDB ):
elif action == 'status_num_inbox': result = self._DoStatusNumInbox( c, *args, **kwargs )
elif action == 'subscriptions': result = self._GetSubscriptions( c, *args, **kwargs )
elif action == 'tag_service_precedence': result = self._tag_service_precedence
elif action == 'tag_parents': result = self._GetTagParents( c, *args, **kwargs )
elif action == 'tag_siblings': result = self._GetTagSiblings( c, *args, **kwargs )
elif action == 'thumbnail': result = self._GetThumbnail( *args, **kwargs )
elif action == 'thumbnail_hashes_i_should_have': result = self._GetThumbnailHashesIShouldHave( c, *args, **kwargs )
@ -7489,6 +7679,7 @@ class DB( ServiceDB ):
elif action == 'set_tag_service_precedence': self._SetTagServicePrecedence( c, *args, **kwargs )
elif action == 'subscription': self._SetSubscription( c, *args, **kwargs )
elif action == 'subscriptions': self._SetSubscriptions( c, *args, **kwargs )
elif action == 'tag_parents': self._UpdateTagParents( c, *args, **kwargs )
elif action == 'tag_siblings': self._UpdateTagSiblings( c, *args, **kwargs )
elif action == 'thumbnails': self._AddThumbnails( c, *args, **kwargs )
elif action == 'update': self._AddUpdate( c, *args, **kwargs )

View File

@ -665,6 +665,15 @@ class FrameGUI( ClientGUICommon.Frame ):
self._options[ 'pause_subs_sync' ] = original_pause_status
def _ManageTagParents( self ):
try:
with ClientGUIDialogs.DialogManageTagParents( self ) as dlg: dlg.ShowModal()
except Exception as e: wx.MessageBox( unicode( e ) + traceback.format_exc() )
def _ManageTagServicePrecedence( self ):
try:
@ -752,6 +761,7 @@ class FrameGUI( ClientGUICommon.Frame ):
elif name == 'hentai foundry by artist': new_page = ClientGUIPages.PageImportHentaiFoundryArtist( self._notebook )
elif name == 'hentai foundry by tags': new_page = ClientGUIPages.PageImportHentaiFoundryTags( self._notebook )
elif name == 'giphy': new_page = ClientGUIPages.PageImportGiphy( self._notebook )
elif name == 'newgrounds': new_page = ClientGUIPages.PageImportNewgrounds( self._notebook )
elif name == 'pixiv by artist': new_page = ClientGUIPages.PageImportPixivArtist( self._notebook )
elif name == 'pixiv by tag': new_page = ClientGUIPages.PageImportPixivTag( self._notebook )
elif name == 'tumblr': new_page = ClientGUIPages.PageImportTumblr( self._notebook )
@ -1066,6 +1076,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
elif command == 'manage_server_services': self._ManageServer( data )
elif command == 'manage_services': self._ManageServices()
elif command == 'manage_subscriptions': self._ManageSubscriptions()
elif command == 'manage_tag_parents': self._ManageTagParents()
elif command == 'manage_tag_service_precedence': self._ManageTagServicePrecedence()
elif command == 'manage_tag_siblings': self._ManageTagSiblings()
elif command == 'modify_account': self._ModifyAccount( data )
@ -1340,6 +1351,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
services.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'manage_services' ), p( '&Add, Remove or Edit Services' ), p( 'Edit your services.' ) )
services.AppendSeparator()
services.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'manage_tag_siblings' ), p( '&Manage Tag Siblings' ), p( 'Set certain tags to be automatically replaced with other tags.' ) )
services.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'manage_tag_parents' ), p( '&Manage Tag Parents' ), p( 'Set certain tags to be automatically added with other tags.' ) )
#services.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'manage_tag_service_precedence' ), p( '&Manage Tag Service Precedence' ), p( 'Change the order in which tag repositories\' taxonomies will be added to the database.' ) )
services.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'manage_tag_service_precedence' ), p( '&Manage Tag Service Precedence' ), p( 'Change the order in which tag repositories\' taxonomies will be added to the database.' ) )
services.AppendSeparator()
@ -1661,7 +1673,7 @@ class FramePageChooser( ClientGUICommon.Frame ):
elif menu_keyword == 'download': entries = [ ( 'page_import_url', None ), ( 'page_import_thread_watcher', None ), ( 'menu', 'gallery' ) ]
elif menu_keyword == 'gallery':
entries = [ ( 'page_import_booru', None ), ( 'page_import_gallery', 'giphy' ), ( 'page_import_gallery', 'deviant art by artist' ), ( 'menu', 'hentai foundry' ) ]
entries = [ ( 'page_import_booru', None ), ( 'page_import_gallery', 'giphy' ), ( 'page_import_gallery', 'deviant art by artist' ), ( 'menu', 'hentai foundry' ), ( 'page_import_gallery', 'newgrounds' ) ]
( id, password ) = wx.GetApp().Read( 'pixiv_account' )
@ -2262,7 +2274,7 @@ class FrameReviewServicesServicePanel( wx.ScrolledWindow ):
def EventServiceReset( self, event ):
message = 'This will remove all cached information for ' + self._service_identifier.GetName() + ' from the database. It will take time to resynchronise.'
message = 'This will remove all cached information for ' + self._service_identifier.GetName() + ' from the database. It will take a minute for the database to finish the operation, during which time the gui may freeze.'
with ClientGUIDialogs.DialogYesNo( self, message ) as dlg:

View File

@ -255,7 +255,7 @@ class Canvas():
def _GetInfoString( self ):
info_string = self._current_media.GetPrettyInfo()
info_string = self._current_media.GetPrettyInfo() + ' | ' + self._current_media.GetPrettyAge()
return info_string
@ -641,7 +641,7 @@ class CanvasFullscreenMediaList( ClientGUIMixins.ListeningMediaList, Canvas, Cli
def _GetInfoString( self ):
info_string = self._current_media.GetPrettyInfo() + ' ' + HC.ConvertZoomToPercentage( self._current_zoom )
info_string = self._current_media.GetPrettyInfo() + ' | ' + HC.ConvertZoomToPercentage( self._current_zoom ) + ' | ' + self._current_media.GetPrettyAge()
return info_string
@ -1056,6 +1056,8 @@ class CanvasFullscreenMediaListBrowser( CanvasFullscreenMediaList ):
elif event.KeyCode in ( ord( '-' ), wx.WXK_SUBTRACT, wx.WXK_NUMPAD_SUBTRACT ): self._ZoomOut()
elif event.KeyCode == ord( 'Z' ): self._ZoomSwitch()
elif event.KeyCode in ( wx.WXK_RETURN, wx.WXK_NUMPAD_ENTER, wx.WXK_ESCAPE ): self.EventClose( event )
elif event.KeyCode == ord( 'C' ) and event.CmdDown():
with wx.BusyCursor(): wx.GetApp().Write( 'copy_files', ( self._current_media.GetHash(), ) )
else:
( modifier, key ) = HC.GetShortcutFromEvent( event )
@ -1088,6 +1090,8 @@ class CanvasFullscreenMediaListBrowser( CanvasFullscreenMediaList ):
( command, data ) = action
if command == 'archive': self._Archive()
elif command == 'copy_files':
with wx.BusyCursor(): wx.GetApp().Write( 'copy_files', ( self._current_media.GetHash(), ) )
elif command == 'copy_local_url': self._CopyLocalUrlToClipboard()
elif command == 'copy_path': self._CopyPathToClipboard()
elif command == 'delete': self._Delete()
@ -1196,8 +1200,13 @@ class CanvasFullscreenMediaListBrowser( CanvasFullscreenMediaList ):
menu.AppendSeparator()
menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_path' ) , 'copy path' )
menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_local_url' ) , 'copy local url' )
copy_menu = wx.Menu()
copy_menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_files' ) , 'file' )
copy_menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_path' ) , 'path' )
copy_menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_local_url' ) , 'local url' )
menu.AppendMenu( CC.ID_NULL, 'copy', copy_menu )
menu.AppendSeparator()
@ -1379,6 +1388,8 @@ class CanvasFullscreenMediaListCustomFilter( CanvasFullscreenMediaList ):
elif event.KeyCode in ( ord( '-' ), wx.WXK_SUBTRACT, wx.WXK_NUMPAD_SUBTRACT ): self._ZoomOut()
elif event.KeyCode == ord( 'Z' ): self._ZoomSwitch()
elif event.KeyCode in ( wx.WXK_RETURN, wx.WXK_NUMPAD_ENTER, wx.WXK_ESCAPE ): self.EventClose( event )
elif event.KeyCode == ord( 'C' ) and event.CmdDown():
with wx.BusyCursor(): wx.GetApp().Write( 'copy_files', ( self._current_media.GetHash(), ) )
else: event.Skip()
@ -1399,6 +1410,8 @@ class CanvasFullscreenMediaListCustomFilter( CanvasFullscreenMediaList ):
( command, data ) = action
if command == 'archive': self._Archive()
elif command == 'copy_files':
with wx.BusyCursor(): wx.GetApp().Write( 'copy_files', ( self._current_media.GetHash(), ) )
elif command == 'copy_local_url': self._CopyLocalUrlToClipboard()
elif command == 'copy_path': self._CopyPathToClipboard()
elif command == 'delete': self._Delete()
@ -1486,8 +1499,13 @@ class CanvasFullscreenMediaListCustomFilter( CanvasFullscreenMediaList ):
menu.AppendSeparator()
menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_path' ) , 'copy path' )
menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_local_url' ) , 'copy local url' )
copy_menu = wx.Menu()
copy_menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_files' ) , 'file' )
copy_menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_path' ) , 'path' )
copy_menu.Append( CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'copy_local_url' ) , 'local url' )
menu.AppendMenu( CC.ID_NULL, 'copy', copy_menu )
menu.AppendSeparator()
@ -1587,6 +1605,8 @@ class CanvasFullscreenMediaListFilter( CanvasFullscreenMediaList ):
except: wx.MessageBox( traceback.format_exc() )
self._current_media = self._GetFirst() # so the pubsub on close is better
CanvasFullscreenMediaList.EventClose( self, event )
@ -1620,6 +1640,8 @@ class CanvasFullscreenMediaListFilter( CanvasFullscreenMediaList ):
elif event.KeyCode == wx.WXK_BACK: self.EventBack( event )
elif event.KeyCode in ( wx.WXK_RETURN, wx.WXK_NUMPAD_ENTER, wx.WXK_ESCAPE ): self.EventClose( event )
elif event.KeyCode in ( wx.WXK_DELETE, wx.WXK_NUMPAD_DELETE ): self.EventDelete( event )
elif event.KeyCode == ord( 'C' ) and event.CmdDown():
with wx.BusyCursor(): wx.GetApp().Write( 'copy_files', ( self._current_media.GetHash(), ) )
elif not event.ShiftDown() and event.KeyCode in ( wx.WXK_UP, wx.WXK_NUMPAD_UP ): self.EventSkip( event )
else:
@ -2375,6 +2397,8 @@ class RatingsFilterFrameNumerical( ClientGUICommon.Frame ):
elif event.KeyCode in ( wx.WXK_RIGHT, wx.WXK_NUMPAD_RIGHT ): self._ProcessAction( 'right' )
elif event.KeyCode == wx.WXK_BACK: self._GoBack()
elif event.KeyCode in ( wx.WXK_RETURN, wx.WXK_NUMPAD_ENTER, wx.WXK_ESCAPE ): self.EventClose( event )
elif event.KeyCode == ord( 'C' ) and event.CmdDown():
with wx.BusyCursor(): wx.GetApp().Write( 'copy_files', ( self._current_media.GetHash(), ) )
else: event.Skip()
@ -2683,6 +2707,8 @@ class RatingsFilterFrameNumerical( ClientGUICommon.Frame ):
if event.KeyCode in ( ord( '+' ), wx.WXK_ADD, wx.WXK_NUMPAD_ADD ): self._ZoomIn()
elif event.KeyCode in ( ord( '-' ), wx.WXK_SUBTRACT, wx.WXK_NUMPAD_SUBTRACT ): self._ZoomOut()
elif event.KeyCode == ord( 'Z' ): self._ZoomSwitch()
elif event.KeyCode == ord( 'C' ) and event.CmdDown():
with wx.BusyCursor(): wx.GetApp().Write( 'copy_files', ( self._current_media.GetHash(), ) )
else: self.GetParent().ProcessEvent( event )

View File

@ -252,10 +252,8 @@ class AutoCompleteDropdown( wx.TextCtrl ):
num_chars = len( self.GetValue() )
if num_chars == 0: lag = 0
elif num_chars == 1: lag = 400
elif num_chars == 2: lag = 200
else: lag = 100
else: lag = 150
#lag = 0
self._lag_timer.Start( lag, wx.TIMER_ONE_SHOT )
@ -397,7 +395,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
self.Bind( wx.EVT_MENU, self.EventMenu )
def _InitCachedResults( self ): return CC.AutocompleteMatchesCounted( {} )
def _InitCachedResults( self ): return CC.AutocompleteMatchesPredicates( HC.LOCAL_FILE_SERVICE_IDENTIFIER, [] )
def _InitDropDownList( self ): return TagsBoxActiveOnly( self._dropdown_window, self.BroadcastChoice )
@ -636,7 +634,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
tags_to_count = collections.Counter( absolutely_all_tags_flat )
self._cached_results = CC.AutocompleteMatchesPredicates( [ HC.Predicate( HC.PREDICATE_TYPE_TAG, ( operator, tag ), count ) for ( tag, count ) in tags_to_count.items() ] )
self._cached_results = CC.AutocompleteMatchesPredicates( self._tag_service_identifier, [ HC.Predicate( HC.PREDICATE_TYPE_TAG, ( operator, tag ), count ) for ( tag, count ) in tags_to_count.items() ] )
@ -705,15 +703,17 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
def _BroadcastChoice( self, predicate ):
if predicate is None: broadcast = None
if predicate is None: self._chosen_tag_callable( None )
else:
( operator, tag ) = predicate.GetValue()
broadcast = tag
parent_manager = wx.GetApp().GetTagParentsManager()
parents = parent_manager.GetParents( self._tag_service_identifier, tag )
self._chosen_tag_callable( tag, parents )
self._chosen_tag_callable( broadcast )
def _GenerateMatches( self ):
@ -753,24 +753,23 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
half_complete_tag = search_text
# this bit obviously now needs an overhaul; we want to change to broader search domains automatically, based on what the user has selected
# (and hopefully show that in the buttons, temporarily)
if len( half_complete_tag ) >= num_first_letters:
if must_do_a_search or self._first_letters == '' or not half_complete_tag.startswith( self._first_letters ):
self._first_letters = half_complete_tag
self._cached_results = wx.GetApp().Read( 'autocomplete_tags', file_service_identifier = self._file_service_identifier, tag_service_identifier = self._tag_service_identifier, half_complete_tag = search_text, do_siblings_collapse = False )
self._cached_results = wx.GetApp().Read( 'autocomplete_tags', file_service_identifier = self._file_service_identifier, tag_service_identifier = self._tag_service_identifier, half_complete_tag = search_text, collapse = False )
matches = self._cached_results.GetMatches( half_complete_tag )
else: matches = []
# now do the 'put whatever they typed in at the top, whether it has count or not'
# do the 'put whatever they typed in at the top, whether it has count or not'
# now with sibling support!
# and parent support!
# this is getting pretty ugly, and I should really move it into the matches processing, I think!
top_predicates = []
@ -784,14 +783,42 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
for predicate in top_predicates:
parents = []
try:
predicate = matches[ matches.index( predicate ) ]
index = matches.index( predicate )
predicate = matches[ index ]
matches.remove( predicate )
except: pass
while matches[ index ].GetPredicateType() == HC.PREDICATE_TYPE_PARENT:
parent = matches[ index ]
matches.remove( parent )
parents.append( parent )
except:
if predicate.GetPredicateType() == HC.PREDICATE_TYPE_TAG:
tag = predicate.GetTag()
parents_manager = wx.GetApp().GetTagParentsManager()
raw_parents = parents_manager.GetParents( self._tag_service_identifier, tag )
parents = [ HC.Predicate( HC.PREDICATE_TYPE_PARENT, raw_parent, None ) for raw_parent in raw_parents ]
parents.reverse()
for parent in parents: matches.insert( 0, parent )
matches.insert( 0, predicate )
@ -2792,6 +2819,7 @@ class TagsBox( ListBox ):
if namespace.startswith( '(+) ' ): namespace = namespace[4:]
if namespace.startswith( '(-) ' ): namespace = namespace[4:]
if namespace.startswith( '(X) ' ): namespace = namespace[4:]
if namespace.startswith( ' ' ): namespace = namespace[4:]
if namespace in namespace_colours: ( r, g, b ) = namespace_colours[ namespace ]
else: ( r, g, b ) = namespace_colours[ None ]
@ -2814,6 +2842,37 @@ class TagsBoxActiveOnly( TagsBox ):
def _Activate( self, s, term ): self._callable( term )
def _Select( self, index ):
if index is not None:
if self._current_selected_index is None: direction = 1
elif index - self._current_selected_index in ( -1, 1 ): direction = index - self._current_selected_index
else: direction = 1
if index == -1 or index > len( self._ordered_strings ): index = len( self._ordered_strings ) - 1
elif index == len( self._ordered_strings ) or index < -1: index = 0
s = self._ordered_strings[ index ]
new_term = self._strings_to_terms[ s ]
while new_term.GetPredicateType() == HC.PREDICATE_TYPE_PARENT:
index += direction
if index == -1 or index > len( self._ordered_strings ): index = len( self._ordered_strings ) - 1
elif index == len( self._ordered_strings ) or index < -1: index = 0
s = self._ordered_strings[ index ]
new_term = self._strings_to_terms[ s ]
ListBox._Select( self, index )
def SetPredicates( self, predicates ):
# need to do a clever compare, since normal predicate compare doesn't take count into account

View File

@ -661,7 +661,7 @@ class DialogInputCustomFilterAction( Dialog ):
return ( ( pretty_modifier, pretty_key, pretty_service_identifier, self._pretty_action ), ( modifier, key, self._service_identifier, self._action ) )
def SetTag( self, tag ): self._tag_value.SetValue( tag )
def SetTag( self, tag, parents = [] ): self._tag_value.SetValue( tag )
class DialogInputNamespaceRegex( Dialog ):
@ -3453,23 +3453,30 @@ class DialogManage4chanPass( Dialog ):
token = self._token.GetValue()
pin = self._pin.GetValue()
form_fields = {}
form_fields[ 'act' ] = 'do_login'
form_fields[ 'id' ] = token
form_fields[ 'pin' ] = pin
form_fields[ 'long_login' ] = 'yes'
( ct, body ) = CC.GenerateMultipartFormDataCTAndBodyFromDict( form_fields )
headers = {}
headers[ 'Content-Type' ] = ct
connection = HC.AdvancedHTTPConnection( url = 'https://sys.4chan.org/', accept_cookies = True )
response = connection.request( 'POST', '/auth', headers = headers, body = body )
self._timeout = int( time.time() ) + 365 * 24 * 3600
if token == '' and pin == '':
self._timeout = 0
else:
form_fields = {}
form_fields[ 'act' ] = 'do_login'
form_fields[ 'id' ] = token
form_fields[ 'pin' ] = pin
form_fields[ 'long_login' ] = 'yes'
( ct, body ) = CC.GenerateMultipartFormDataCTAndBodyFromDict( form_fields )
headers = {}
headers[ 'Content-Type' ] = ct
connection = HC.AdvancedHTTPConnection( url = 'https://sys.4chan.org/', accept_cookies = True )
response = connection.request( 'POST', '/auth', headers = headers, body = body )
self._timeout = int( time.time() ) + 365 * 24 * 3600
wx.GetApp().Write( '4chan_pass', token, pin, self._timeout )
@ -6611,6 +6618,9 @@ class DialogManageSubscriptions( Dialog ):
self._giphy = ClientGUICommon.ListBook( self._listbook )
self._giphy.Bind( wx.EVT_NOTEBOOK_PAGE_CHANGING, self.EventServiceChanging )
self._newgrounds = ClientGUICommon.ListBook( self._listbook )
self._newgrounds.Bind( wx.EVT_NOTEBOOK_PAGE_CHANGING, self.EventServiceChanging )
self._pixiv = ClientGUICommon.ListBook( self._listbook )
self._pixiv.Bind( wx.EVT_NOTEBOOK_PAGE_CHANGING, self.EventServiceChanging )
@ -6626,6 +6636,7 @@ class DialogManageSubscriptions( Dialog ):
types_to_listbooks[ HC.SITE_DOWNLOAD_TYPE_PIXIV ] = self._pixiv
types_to_listbooks[ HC.SITE_DOWNLOAD_TYPE_BOORU ] = self._booru
types_to_listbooks[ HC.SITE_DOWNLOAD_TYPE_TUMBLR ] = self._tumblr
types_to_listbooks[ HC.SITE_DOWNLOAD_TYPE_NEWGROUNDS ] = self._newgrounds
for ( site_download_type, name, query_type, query, frequency_type, frequency_number, advanced_tag_options, advanced_import_options, last_checked, url_cache, paused ) in self._original_subscriptions:
@ -6639,6 +6650,7 @@ class DialogManageSubscriptions( Dialog ):
self._listbook.AddPage( self._deviant_art, 'deviant art' )
self._listbook.AddPage( self._hentai_foundry, 'hentai foundry' )
self._listbook.AddPage( self._giphy, 'giphy' )
self._listbook.AddPage( self._newgrounds, 'newgrounds' )
self._listbook.AddPage( self._pixiv, 'pixiv' )
self._listbook.AddPage( self._booru, 'booru' )
self._listbook.AddPage( self._tumblr, 'tumblr' )
@ -6745,6 +6757,7 @@ class DialogManageSubscriptions( Dialog ):
elif subscription_listbook == self._pixiv: site_download_type = HC.SITE_DOWNLOAD_TYPE_PIXIV
elif subscription_listbook == self._booru: site_download_type = HC.SITE_DOWNLOAD_TYPE_BOORU
elif subscription_listbook == self._tumblr: site_download_type = HC.SITE_DOWNLOAD_TYPE_TUMBLR
elif subscription_listbook == self._newgrounds: site_download_type = HC.SITE_DOWNLOAD_TYPE_NEWGROUNDS
if site_download_type == HC.SITE_DOWNLOAD_TYPE_PIXIV:
@ -6758,7 +6771,7 @@ class DialogManageSubscriptions( Dialog ):
if site_download_type in ( HC.SITE_DOWNLOAD_TYPE_DEVIANT_ART, HC.SITE_DOWNLOAD_TYPE_TUMBLR ): query_type = 'artist'
if site_download_type in ( HC.SITE_DOWNLOAD_TYPE_DEVIANT_ART, HC.SITE_DOWNLOAD_TYPE_TUMBLR, HC.SITE_DOWNLOAD_TYPE_NEWGROUNDS ): query_type = 'artist'
else: query_type = 'tags'
if site_download_type == HC.SITE_DOWNLOAD_TYPE_BOORU: query_type = ( '', query_type )
@ -6862,6 +6875,7 @@ class DialogManageSubscriptions( Dialog ):
all_pages.extend( self._pixiv.GetNameToPageDict().values() )
all_pages.extend( self._booru.GetNameToPageDict().values() )
all_pages.extend( self._tumblr.GetNameToPageDict().values() )
all_pages.extend( self._newgrounds.GetNameToPageDict().values() )
subscriptions = [ page.GetInfo() for page in all_pages ]
@ -6928,6 +6942,7 @@ class DialogManageSubscriptions( Dialog ):
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_HENTAI_FOUNDRY: services_listbook = self._hentai_foundry
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_PIXIV: services_listbook = self._pixiv
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_TUMBLR: services_listbook = self._tumblr
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_NEWGROUNDS: services_listbook = self._newgrounds
self._listbook.SelectPage( services_listbook )
@ -6995,7 +7010,7 @@ class DialogManageSubscriptions( Dialog ):
self._query_type = ClientGUICommon.RadioBox( self._query_panel, 'query type', ( ( 'artist', 'artist' ), ( 'tags', 'tags' ) ) )
if site_download_type in ( HC.SITE_DOWNLOAD_TYPE_BOORU, HC.SITE_DOWNLOAD_TYPE_DEVIANT_ART, HC.SITE_DOWNLOAD_TYPE_GIPHY, HC.SITE_DOWNLOAD_TYPE_TUMBLR ): self._query_type.Hide()
if site_download_type in ( HC.SITE_DOWNLOAD_TYPE_BOORU, HC.SITE_DOWNLOAD_TYPE_DEVIANT_ART, HC.SITE_DOWNLOAD_TYPE_GIPHY, HC.SITE_DOWNLOAD_TYPE_TUMBLR, HC.SITE_DOWNLOAD_TYPE_NEWGROUNDS ): self._query_type.Hide()
self._frequency_number = wx.SpinCtrl( self._query_panel )
@ -7016,6 +7031,7 @@ class DialogManageSubscriptions( Dialog ):
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_HENTAI_FOUNDRY: namespaces = [ 'creator', 'title', '' ]
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_PIXIV: namespaces = [ 'creator', 'title', '' ]
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_TUMBLR: namespaces = [ '' ]
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_NEWGROUNDS: namespaces = [ 'creator', 'title', '' ]
self._advanced_tag_options = ClientGUICommon.AdvancedTagOptions( self, 'send tags to ', namespaces )
@ -7091,7 +7107,7 @@ class DialogManageSubscriptions( Dialog ):
self._query_type.SetSelection( initial_index )
if site_download_type in ( HC.SITE_DOWNLOAD_TYPE_BOORU, HC.SITE_DOWNLOAD_TYPE_DEVIANT_ART, HC.SITE_DOWNLOAD_TYPE_GIPHY, HC.SITE_DOWNLOAD_TYPE_TUMBLR ): self._query_type.Hide()
if site_download_type in ( HC.SITE_DOWNLOAD_TYPE_BOORU, HC.SITE_DOWNLOAD_TYPE_DEVIANT_ART, HC.SITE_DOWNLOAD_TYPE_GIPHY, HC.SITE_DOWNLOAD_TYPE_TUMBLR, HC.SITE_DOWNLOAD_TYPE_NEWGROUNDS ): self._query_type.Hide()
self._frequency_number.SetValue( frequency_number )
@ -7172,7 +7188,332 @@ class DialogManageSubscriptions( Dialog ):
class DialogManageTagParents( Dialog ):
def __init__( self, parent ):
def InitialiseControls():
self._tag_repositories = ClientGUICommon.ListBook( self )
self._tag_repositories.Bind( wx.EVT_NOTEBOOK_PAGE_CHANGED, self.EventServiceChanged )
#services = wx.GetApp().Read( 'services', ( HC.TAG_REPOSITORY, ) )
#for service in services:
#
# account = service.GetAccount()
#
# if account.HasPermission( HC.POST_DATA ):
#
# service_identifier = service.GetServiceIdentifier()
#
# page_info = ( self._Panel, ( self._tag_repositories, service_identifier, paths ), {} )
#
# name = service_identifier.GetName()
#
# self._tag_repositories.AddPage( page_info, name )
#
#
page = self._Panel( self._tag_repositories, HC.LOCAL_TAG_SERVICE_IDENTIFIER )
name = HC.LOCAL_TAG_SERVICE_IDENTIFIER.GetName()
self._tag_repositories.AddPage( page, name )
#default_tag_repository = self._options[ 'default_tag_repository' ]
#self._tag_repositories.Select( default_tag_repository.GetName() )
self._ok_button = wx.Button( self, label='ok' )
self._ok_button.Bind( wx.EVT_BUTTON, self.EventOK )
self._ok_button.SetForegroundColour( ( 0, 128, 0 ) )
self._close_button = wx.Button( self, id = wx.ID_CANCEL, label='cancel' )
self._close_button.Bind( wx.EVT_BUTTON, self.EventCancel )
self._close_button.SetForegroundColour( ( 128, 0, 0 ) )
def InitialisePanel():
buttons = wx.BoxSizer( wx.HORIZONTAL )
buttons.AddF( self._ok_button, FLAGS_SMALL_INDENT )
buttons.AddF( self._close_button, FLAGS_SMALL_INDENT )
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.AddF( self._tag_repositories, FLAGS_EXPAND_BOTH_WAYS )
vbox.AddF( buttons, FLAGS_BUTTON_SIZERS )
self.SetSizer( vbox )
self.SetInitialSize( ( 450, 680 ) )
Dialog.__init__( self, parent, 'tag parents' )
InitialiseControls()
InitialisePanel()
interested_actions = [ 'set_search_focus' ]
entries = []
for ( modifier, key_dict ) in self._options[ 'shortcuts' ].items(): entries.extend( [ ( modifier, key, CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( action ) ) for ( key, action ) in key_dict.items() if action in interested_actions ] )
self.SetAcceleratorTable( wx.AcceleratorTable( entries ) )
self.Bind( wx.EVT_MENU, self.EventMenu )
def _SetSearchFocus( self ):
page = self._tag_repositories.GetCurrentPage()
page.SetTagBoxFocus()
def EventCancel( self, event ): self.EndModal( wx.ID_CANCEL )
def EventMenu( self, event ):
action = CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetAction( event.GetId() )
if action is not None:
try:
( command, data ) = action
if command == 'set_search_focus': self._SetSearchFocus()
else: event.Skip()
except Exception as e:
wx.MessageBox( unicode( e ) )
wx.MessageBox( traceback.format_exc() )
def EventOK( self, event ):
edit_log = []
try:
for page in self._tag_repositories.GetNameToPageDict().values(): edit_log.append( page.GetChanges() )
if len( edit_log ) > 0: wx.GetApp().Write( 'tag_parents', edit_log )
except Exception as e: wx.MessageBox( 'Saving tag parent changes to DB raised this error: ' + unicode( e ) )
self.EndModal( wx.ID_OK )
def EventServiceChanged( self, event ):
page = self._tag_repositories.GetCurrentPage()
wx.CallAfter( page.SetTagBoxFocus )
class _Panel( wx.Panel ):
def __init__( self, parent, service_identifier ):
def InitialiseControls():
self._tag_parents = ClientGUICommon.SaneListCtrl( self, 250, [ ( 'child', 160 ), ( 'parent', -1 ) ] )
for ( child, parent ) in self._original_pairs: self._tag_parents.Append( ( child, parent ), ( child, parent ) )
self._tag_parents.Bind( wx.EVT_LIST_ITEM_SELECTED, self.EventItemSelected )
self._tag_parents.Bind( wx.EVT_LIST_ITEM_DESELECTED, self.EventItemSelected )
self._child_text = wx.StaticText( self )
self._parent_text = wx.StaticText( self )
self._child_input = ClientGUICommon.AutoCompleteDropdownTagsWrite( self, self.SetChild, HC.LOCAL_FILE_SERVICE_IDENTIFIER, service_identifier )
self._parent_input = ClientGUICommon.AutoCompleteDropdownTagsWrite( self, self.SetParent, HC.LOCAL_FILE_SERVICE_IDENTIFIER, service_identifier )
self._add = wx.Button( self, label = 'add' )
self._add.Bind( wx.EVT_BUTTON, self.EventAdd )
self._add.Disable()
self._remove = wx.Button( self, label = 'remove' )
self._remove.Bind( wx.EVT_BUTTON, self.EventRemove )
self._remove.Disable()
def InitialisePanel():
button_box = wx.BoxSizer( wx.HORIZONTAL )
button_box.AddF( self._add, FLAGS_MIXED )
button_box.AddF( self._remove, FLAGS_MIXED )
text_box = wx.BoxSizer( wx.HORIZONTAL )
text_box.AddF( self._child_text, FLAGS_EXPAND_BOTH_WAYS )
text_box.AddF( self._parent_text, FLAGS_EXPAND_BOTH_WAYS )
input_box = wx.BoxSizer( wx.HORIZONTAL )
input_box.AddF( self._child_input, FLAGS_EXPAND_BOTH_WAYS )
input_box.AddF( self._parent_input, FLAGS_EXPAND_BOTH_WAYS )
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.AddF( self._tag_parents, FLAGS_EXPAND_BOTH_WAYS )
vbox.AddF( button_box, FLAGS_BUTTON_SIZERS )
vbox.AddF( text_box, FLAGS_EXPAND_SIZER_PERPENDICULAR )
vbox.AddF( input_box, FLAGS_EXPAND_SIZER_PERPENDICULAR )
self.SetSizer( vbox )
wx.Panel.__init__( self, parent )
self._service_identifier = service_identifier
self._added = set()
self._removed = set()
self._original_pairs = [ tuple( pair ) for pair in wx.GetApp().Read( 'tag_parents', service_identifier ) ]
self._current_pairs = set( self._original_pairs )
self._current_parent = None
self._current_child = None
InitialiseControls()
InitialisePanel()
def _SetButtonStatus( self ):
all_selected = self._tag_parents.GetAllSelected()
if len( all_selected ) == 0: self._remove.Disable()
else: self._remove.Enable()
if self._current_parent is None or self._current_child is None: self._add.Disable()
else: self._add.Enable()
def EventAdd( self, event ):
pair = ( self._current_child, self._current_parent )
if pair in self._current_pairs:
wx.MessageBox( 'That relationship already exists!' )
return
current_children = { child for ( child, parent ) in self._current_pairs }
# test for loops
if self._current_parent in current_children:
d = dict( self._current_pairs )
next_parent = self._current_parent
while next_parent in d:
next_parent = d[ next_parent ]
if next_parent == self._current_child:
wx.MessageBox( 'Adding that pair would create a loop!' )
return
# if we got here, we are great to do it
self._added.add( pair )
self._removed.discard( pair )
self._current_pairs.add( pair )
self._tag_parents.Append( pair, pair )
self.SetChild( None )
self.SetParent( None )
def EventItemSelected( self, event ):
self._SetButtonStatus()
def EventRemove( self, event ):
all_selected = self._tag_parents.GetAllSelected()
for selected in all_selected:
pair = self._tag_parents.GetClientData( selected )
self._removed.add( pair )
self._added.discard( pair )
self._current_pairs.discard( pair )
self._tag_parents.RemoveAllSelected()
def GetChanges( self ):
edit_log = []
if self._service_identifier == HC.LOCAL_TAG_SERVICE_IDENTIFIER:
edit_log += [ ( HC.ADD, pair ) for pair in self._added ]
edit_log += [ ( HC.DELETE, pair ) for pair in self._removed ]
return ( self._service_identifier, edit_log )
def SetParent( self, tag, parents = [] ):
if tag is not None and tag == self._current_child: self.SetChild( None )
self._current_parent = tag
if tag is None: self._parent_text.SetLabel( '' )
else: self._parent_text.SetLabel( tag )
self._SetButtonStatus()
def SetChild( self, tag, parents = [] ):
if tag is not None and tag == self._current_parent: self.SetParent( None )
self._current_child = tag
if tag is None: self._child_text.SetLabel( '' )
else: self._child_text.SetLabel( tag )
self._SetButtonStatus()
def SetTagBoxFocus( self ): self._child_input.SetFocus()
class DialogManageTagSiblings( Dialog ):
def __init__( self, parent ):
@ -7401,11 +7742,11 @@ class DialogManageTagSiblings( Dialog ):
return
self._current_olds = { old for ( old, new ) in self._current_pairs }
current_olds = { old for ( old, new ) in self._current_pairs }
# test for ambiguity
if self._current_old in self._current_olds:
if self._current_old in current_olds:
wx.MessageBox( 'There already is a relationship set for the tag ' + self._current_new + '.' )
@ -7414,7 +7755,7 @@ class DialogManageTagSiblings( Dialog ):
# test for loops
if self._current_new in self._current_olds:
if self._current_new in current_olds:
d = dict( self._current_pairs )
@ -7481,7 +7822,7 @@ class DialogManageTagSiblings( Dialog ):
return ( self._service_identifier, edit_log )
def SetNew( self, tag ):
def SetNew( self, tag, parents = [] ):
if tag is not None and tag == self._current_old: self.SetOld( None )
@ -7493,7 +7834,7 @@ class DialogManageTagSiblings( Dialog ):
self._SetButtonStatus()
def SetOld( self, tag ):
def SetOld( self, tag, parents = [] ):
if tag is not None and tag == self._current_new: self.SetNew( None )
@ -7859,100 +8200,96 @@ class DialogManageTags( Dialog ):
InitialisePanel()
def AddTag( self, tag ):
def _AddTag( self, tag, only_add = False ):
if tag is None: wx.PostEvent( self, wx.CommandEvent( commandType = wx.wxEVT_COMMAND_MENU_SELECTED, winid = CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'ok' ) ) )
if self._i_am_local_tag_service:
if tag in self._pending_tags:
if only_add: return
self._pending_tags.remove( tag )
self._tags_box.RescindPend( tag )
elif tag in self._petitioned_tags:
self._petitioned_tags.remove( tag )
self._tags_box.RescindPetition( tag )
elif tag in self._current_tags:
if only_add: return
self._petitioned_tags.append( tag )
self._tags_box.PetitionTag( tag )
else:
self._pending_tags.append( tag )
self._tags_box.PendTag( tag )
self._edit_log = []
self._edit_log.extend( [ ( HC.CONTENT_UPDATE_ADD, tag ) for tag in self._pending_tags ] )
self._edit_log.extend( [ ( HC.CONTENT_UPDATE_DELETE, tag ) for tag in self._petitioned_tags ] )
else:
if self._i_am_local_tag_service:
if tag in self._pending_tags:
if tag in self._pending_tags:
if only_add: return
self._pending_tags.remove( tag )
self._tags_box.RescindPend( tag )
self._edit_log.append( ( HC.CONTENT_UPDATE_RESCIND_PENDING, tag ) )
elif tag in self._petitioned_tags:
self._petitioned_tags.remove( tag )
self._tags_box.RescindPetition( tag )
self._edit_log.append( ( HC.CONTENT_UPDATE_RESCIND_PETITION, tag ) )
elif tag in self._current_tags:
if only_add: return
if self._account.HasPermission( HC.RESOLVE_PETITIONS ):
self._pending_tags.remove( tag )
self._tags_box.RescindPend( tag )
elif tag in self._petitioned_tags:
self._petitioned_tags.remove( tag )
self._tags_box.RescindPetition( tag )
elif tag in self._current_tags:
self._edit_log.append( ( HC.CONTENT_UPDATE_PETITION, ( tag, 'admin' ) ) )
self._petitioned_tags.append( tag )
self._tags_box.PetitionTag( tag )
else:
elif self._account.HasPermission( HC.POST_PETITIONS ):
self._pending_tags.append( tag )
message = 'Enter a reason for this tag to be removed. A janitor will review your petition.'
self._tags_box.PendTag( tag )
self._edit_log = []
self._edit_log.extend( [ ( HC.CONTENT_UPDATE_ADD, tag ) for tag in self._pending_tags ] )
self._edit_log.extend( [ ( HC.CONTENT_UPDATE_DELETE, tag ) for tag in self._petitioned_tags ] )
else:
if tag in self._pending_tags:
self._pending_tags.remove( tag )
self._tags_box.RescindPend( tag )
self._edit_log.append( ( HC.CONTENT_UPDATE_RESCIND_PENDING, tag ) )
elif tag in self._petitioned_tags:
self._petitioned_tags.remove( tag )
self._tags_box.RescindPetition( tag )
self._edit_log.append( ( HC.CONTENT_UPDATE_RESCIND_PETITION, tag ) )
elif tag in self._current_tags:
if self._account.HasPermission( HC.RESOLVE_PETITIONS ):
with wx.TextEntryDialog( self, message ) as dlg:
self._edit_log.append( ( HC.CONTENT_UPDATE_PETITION, ( tag, 'admin' ) ) )
self._petitioned_tags.append( tag )
self._tags_box.PetitionTag( tag )
elif self._account.HasPermission( HC.POST_PETITIONS ):
message = 'Enter a reason for this tag to be removed. A janitor will review your petition.'
with wx.TextEntryDialog( self, message ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
if dlg.ShowModal() == wx.ID_OK:
self._edit_log.append( ( HC.CONTENT_UPDATE_PETITION, ( tag, dlg.GetValue() ) ) )
self._petitioned_tags.append( tag )
self._tags_box.PetitionTag( tag )
self._edit_log.append( ( HC.CONTENT_UPDATE_PETITION, ( tag, dlg.GetValue() ) ) )
self._petitioned_tags.append( tag )
self._tags_box.PetitionTag( tag )
elif tag in self._deleted_tags:
if self._account.HasPermission( HC.RESOLVE_PETITIONS ):
self._edit_log.append( ( HC.CONTENT_UPDATE_PENDING, tag ) )
self._pending_tags.append( tag )
self._tags_box.PendTag( tag )
else:
elif tag in self._deleted_tags:
if self._account.HasPermission( HC.RESOLVE_PETITIONS ):
self._edit_log.append( ( HC.CONTENT_UPDATE_PENDING, tag ) )
@ -7961,6 +8298,25 @@ class DialogManageTags( Dialog ):
self._tags_box.PendTag( tag )
else:
self._edit_log.append( ( HC.CONTENT_UPDATE_PENDING, tag ) )
self._pending_tags.append( tag )
self._tags_box.PendTag( tag )
def AddTag( self, tag, parents = [] ):
if tag is None: wx.PostEvent( self, wx.CommandEvent( commandType = wx.wxEVT_COMMAND_MENU_SELECTED, winid = CC.MENU_EVENT_ID_TO_ACTION_CACHE.GetId( 'ok' ) ) )
else:
self._AddTag( tag )
for parent in parents: self._AddTag( parent, only_add = True )

View File

@ -1626,6 +1626,8 @@ class ManagementPanelImportWithQueueAdvanced( ManagementPanelImportWithQueue ):
self.SetSizer( vbox )
wx.CallAfter( self._new_queue_input.SelectAll )
def _InitExtraVboxElements( self, vbox ): pass
@ -1893,6 +1895,20 @@ class ManagementPanelImportWithQueueAdvancedHentaiFoundryTags( ManagementPanelIm
return ( HydrusDownloading.GetDownloader( HC.SITE_DOWNLOAD_TYPE_HENTAI_FOUNDRY, 'tags', tags, self._advanced_hentai_foundry_options.GetInfo() ), )
class ManagementPanelImportWithQueueAdvancedNewgrounds( ManagementPanelImportWithQueueAdvanced ):
def __init__( self, parent, page, page_key ):
name = 'newgrounds'
namespaces = [ 'creator', 'title', '' ]
ManagementPanelImportWithQueueAdvanced.__init__( self, parent, page, page_key, name, namespaces )
self._new_queue_input.SetValue( 'artist' )
def _GetDownloaders( self, artist ): return ( HydrusDownloading.GetDownloader( HC.SITE_DOWNLOAD_TYPE_NEWGROUNDS, artist ), )
class ManagementPanelImportWithQueueAdvancedPixiv( ManagementPanelImportWithQueueAdvanced ):
def __init__( self, parent, page, page_key ):
@ -2106,10 +2122,13 @@ class ManagementPanelImportThreadWatcher( ManagementPanelImport ):
self._thread_panel.AddF( hbox, FLAGS_EXPAND_SIZER_PERPENDICULAR )
self._thread_panel.AddF( self._thread_pause_button, FLAGS_EXPAND_PERPENDICULAR )
self._advanced_tag_options = ClientGUICommon.AdvancedTagOptions( self, 'send to ', [ 'filename' ] )
self._advanced_import_options = ClientGUICommon.AdvancedImportOptions( self )
vbox.AddF( self._processing_panel, FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._thread_panel, FLAGS_EXPAND_SIZER_PERPENDICULAR )
vbox.AddF( self._advanced_tag_options, FLAGS_EXPAND_PERPENDICULAR )
vbox.AddF( self._advanced_import_options, FLAGS_EXPAND_PERPENDICULAR )
self._MakeCurrentSelectionTagsBox( vbox )
@ -2148,7 +2167,7 @@ class ManagementPanelImportThreadWatcher( ManagementPanelImport ):
posts_list = json_dict[ 'posts' ]
image_infos = [ ( post[ 'md5' ].decode( 'base64' ), str( post[ 'tim' ] ), post[ 'ext' ] ) for post in posts_list if 'md5' in post ]
image_infos = [ ( post[ 'md5' ].decode( 'base64' ), str( post[ 'tim' ] ), post[ 'ext' ], post[ 'filename' ] ) for post in posts_list if 'md5' in post ]
image_infos_i_can_add = [ image_info for image_info in image_infos if image_info not in self._image_infos_already_added ]
@ -2182,7 +2201,7 @@ class ManagementPanelImportThreadWatcher( ManagementPanelImport ):
try:
( md5, image_name, ext ) = queue_object
( md5, image_name, ext, filename ) = queue_object
( status, hash ) = wx.GetApp().Read( 'md5_status', md5 )
@ -2228,7 +2247,11 @@ class ManagementPanelImportThreadWatcher( ManagementPanelImport ):
advanced_import_options = self._advanced_import_options.GetInfo()
service_identifiers_to_tags = {}
advanced_tag_options = self._advanced_tag_options.GetInfo()
tags = [ 'filename:' + filename + ext ]
service_identifiers_to_tags = HydrusDownloading.ConvertTagsToServiceIdentifiersToTags( tags, advanced_tag_options )
wx.CallAfter( self.CALLBACKImportArgs, file, advanced_import_options, service_identifiers_to_tags, url = url )

View File

@ -14,7 +14,6 @@ import wx
# Option Enums
ID_TIMER_WATERFALL = wx.NewId()
ID_TIMER_ANIMATION = wx.NewId()
# Sizer Flags
@ -206,21 +205,10 @@ class MediaPanel( ClientGUIMixins.ListeningMediaList, wx.ScrolledWindow ):
def _DeselectAll( self ):
if len( self._selected_media ) > 0:
for m in self._selected_media: m.Deselect()
self._ReblitMedia( self._selected_media )
self._selected_media = set()
self._SetFocussedMedia( None )
self._shift_focussed_media = None
self._ReblitCanvas()
self._PublishSelectionChange()
self._DeselectSelect( self._sorted_media, [] )
self._SetFocussedMedia( None )
self._shift_focussed_media = None
def _DeselectSelect( self, media_to_deselect, media_to_select ):
@ -553,16 +541,7 @@ class MediaPanel( ClientGUIMixins.ListeningMediaList, wx.ScrolledWindow ):
def _SelectAll( self ):
for media in self._sorted_media: media.Select()
self._selected_media = set( self._sorted_media )
self._ReblitCanvas()
self._PublishSelectionChange()
def _SelectAll( self ): self._DeselectSelect( [], self._sorted_media )
def _SetFocussedMedia( self, media ):
@ -757,15 +736,10 @@ class MediaPanelThumbnails( MediaPanel ):
MediaPanel.__init__( self, parent, page_key, file_service_identifier, predicates, file_query_result )
self._last_animation_frame_time = 0.0
self._num_columns = 1
self._num_rows_in_client_height = 0
self._last_visible_row = 0
self._timer_waterfall = wx.Timer( self, ID_TIMER_WATERFALL )
self._thumbnails_to_waterfall = []
self._timer_animation = wx.Timer( self, ID_TIMER_ANIMATION )
self._thumbnails_being_faded_in = {}
@ -786,7 +760,6 @@ class MediaPanelThumbnails( MediaPanel ):
self.Bind( wx.EVT_MIDDLE_DOWN, self.EventMouseFullScreen )
self.Bind( wx.EVT_PAINT, self.EventPaint )
self.Bind( wx.EVT_SIZE, self.EventResize )
self.Bind( wx.EVT_TIMER, self.EventTimerWaterfall, id = ID_TIMER_WATERFALL )
self.Bind( wx.EVT_TIMER, self.EventTimerAnimation, id = ID_TIMER_ANIMATION )
self.Bind( wx.EVT_KEY_DOWN, self.EventKeyDown )
@ -798,6 +771,7 @@ class MediaPanelThumbnails( MediaPanel ):
HC.pubsub.sub( self, 'NewThumbnails', 'new_thumbnails' )
HC.pubsub.sub( self, 'ThumbnailsResized', 'thumbnail_resize' )
HC.pubsub.sub( self, 'RefreshAcceleratorTable', 'options_updated' )
HC.pubsub.sub( self, 'WaterfallThumbnail', 'waterfall_thumbnail' )
def _BlitThumbnail( self, thumbnail ):
@ -996,16 +970,9 @@ class MediaPanelThumbnails( MediaPanel ):
self._last_visible_row = to_row
self._thumbnails_to_waterfall = thumbnails_to_render_later
self._thumbnails_being_faded_in = {}
hashes_to_waterfall = { thumbnail.GetHash() for thumbnail in self._thumbnails_to_waterfall }
wx.GetApp().GetThumbnailCache().Prefetch( hashes_to_waterfall )
random.shuffle( self._thumbnails_to_waterfall )
if not self._timer_waterfall.IsRunning(): self._timer_waterfall.Start( 20, wx.TIMER_ONE_SHOT )
wx.GetApp().GetThumbnailCache().Waterfall( self._page_key, thumbnails_to_render_later )
@ -1536,75 +1503,68 @@ class MediaPanelThumbnails( MediaPanel ):
def EventTimerAnimation( self, event ):
now = time.clock()
started = time.clock()
if now - self._last_animation_frame_time > 1.0 / 60:
( thumbnail_width, thumbnail_height ) = self._thumbnail_span_dimensions
( start_x, start_y ) = self.GetViewStart()
( x_unit, y_unit ) = self.GetScrollPixelsPerUnit()
( width, height ) = self.GetClientSize()
min_y = ( start_y * y_unit ) - thumbnail_height
max_y = ( start_y * y_unit ) + height + thumbnail_height
dc = self._GetScrolledDC()
all_info = self._thumbnails_being_faded_in.items()
for ( key, ( alpha_bmp, num_frames_rendered ) ) in all_info:
( thumbnail_width, thumbnail_height ) = self._thumbnail_span_dimensions
( bmp, x, y ) = key
( start_x, start_y ) = self.GetViewStart()
( x_unit, y_unit ) = self.GetScrollPixelsPerUnit()
( width, height ) = self.GetClientSize()
min_y = ( start_y * y_unit ) - thumbnail_height
max_y = ( start_y * y_unit ) + height + thumbnail_height
dc = self._GetScrolledDC()
all_info = self._thumbnails_being_faded_in.items()
for ( key, ( alpha_bmp, num_frames_rendered ) ) in all_info:
if num_frames_rendered == 0:
( bmp, x, y ) = key
image = bmp.ConvertToImage()
if num_frames_rendered == 0:
image = bmp.ConvertToImage()
image.InitAlpha()
image = image.AdjustChannels( 1, 1, 1, 0.25 )
alpha_bmp = wx.BitmapFromImage( image, 32 )
image.InitAlpha()
num_frames_rendered += 1
image = image.AdjustChannels( 1, 1, 1, 0.25 )
self._thumbnails_being_faded_in[ key ] = ( alpha_bmp, num_frames_rendered )
if y < min_y or y > max_y or num_frames_rendered == 9:
bmp_to_use = bmp
del self._thumbnails_being_faded_in[ key ]
else:
bmp_to_use = alpha_bmp
dc.DrawBitmap( bmp_to_use, x, y, True )
alpha_bmp = wx.BitmapFromImage( image, 32 )
self._last_animation_frame_time = now
num_frames_rendered += 1
self._thumbnails_being_faded_in[ key ] = ( alpha_bmp, num_frames_rendered )
if y < min_y or y > max_y or num_frames_rendered == 9:
bmp_to_use = bmp
del self._thumbnails_being_faded_in[ key ]
else:
bmp_to_use = alpha_bmp
dc.DrawBitmap( bmp_to_use, x, y, True )
if time.clock() - started > 0.016: break
if len( self._thumbnails_being_faded_in ) > 0: self._timer_animation.Start( 6, wx.TIMER_ONE_SHOT )
finished = time.clock()
def EventTimerWaterfall( self, event ):
( min, max, milliseconds ) = ( 5, 10, 8 )
how_many = random.randint( min, max )
for thumbnail in self._thumbnails_to_waterfall[-how_many:]: self._BlitThumbnail( thumbnail )
self._thumbnails_to_waterfall = self._thumbnails_to_waterfall[:-how_many]
if len( self._thumbnails_to_waterfall ) > 0: self._timer_waterfall.Start( milliseconds, wx.TIMER_ONE_SHOT )
if len( self._thumbnails_being_faded_in ) > 0:
time_this_took_in_ms = ( finished - started ) * 1000
ms = max( 1, int( round( 16.7 - time_this_took_in_ms ) ) )
self._timer_animation.Start( ms, wx.TIMER_ONE_SHOT )
def NewThumbnails( self, hashes ):
@ -1689,6 +1649,13 @@ class MediaPanelThumbnails( MediaPanel ):
self._ReblitCanvas()
def WaterfallThumbnail( self, page_key, thumbnail, thumbnail_bmp ):
thumbnail.SetBmp( thumbnail_bmp )
self._BlitThumbnail( thumbnail )
class Selectable():
def __init__( self ): self._selected = False
@ -1774,18 +1741,9 @@ class Thumbnail( Selectable ):
self._my_dimensions = CC.AddPaddingToDimensions( wx.GetApp().Read( 'options' )[ 'thumbnail_dimensions' ], CC.THUMBNAIL_BORDER * 2 )
def _LoadFromDB( self ):
display_hash = self.GetDisplayMedia().GetHash()
mime = self.GetDisplayMedia().GetMime()
if mime in HC.IMAGES: self._hydrus_bmp = wx.GetApp().GetThumbnailCache().GetThumbnail( display_hash )
elif mime == HC.APPLICATION_FLASH: self._hydrus_bmp = wx.GetApp().GetThumbnailCache().GetFlashThumbnail()
elif mime == HC.APPLICATION_PDF: self._hydrus_bmp = wx.GetApp().GetThumbnailCache().GetPDFThumbnail()
elif mime == HC.VIDEO_FLV: self._hydrus_bmp = wx.GetApp().GetThumbnailCache().GetFLVThumbnail()
else: self._hydrus_bmp = wx.GetApp().GetThumbnailCache().GetNotFoundThumbnail()
def _LoadFromDB( self ): self._hydrus_bmp = wx.GetApp().GetThumbnailCache().GetThumbnail( self )
def Dumped( self, dump_status ): self._dump_status = dump_status
def GetBmp( self ):
@ -1974,8 +1932,6 @@ class Thumbnail( Selectable ):
return bmp
def Dumped( self, dump_status ): self._dump_status = dump_status
def IsLoaded( self ): return self._hydrus_bmp is not None
def ReloadFromDB( self ):
@ -1992,6 +1948,8 @@ class Thumbnail( Selectable ):
self._hydrus_bmp = None
def SetBmp( self, bmp ): self._hydrus_bmp = bmp
class ThumbnailMediaCollection( Thumbnail, ClientGUIMixins.MediaCollection ):
def __init__( self, file_service_identifier, predicates, media_results ):

View File

@ -522,7 +522,7 @@ class MediaCollection( MediaList, Media ):
def GetNumWords( self ): return sum( [ media.GetNumWords() for media in self._sorted_media ] )
def GetPrettyAge( self ): return HC.ConvertTimestampToPrettyAge( self._timestamp )
def GetPrettyAge( self ): return 'imported ' + HC.ConvertTimestampToPrettyAgo( self._timestamp )
def GetPrettyInfo( self ):
@ -637,7 +637,7 @@ class MediaSingleton( Media ):
else: return timestamp
def GetPrettyAge( self ): return HC.ConvertTimestampToPrettyAge( self._media_result.GetTimestamp() )
def GetPrettyAge( self ): return 'imported ' + HC.ConvertTimestampToPrettyAgo( self._media_result.GetTimestamp() )
def GetPrettyInfo( self ):

View File

@ -313,6 +313,15 @@ class PageImportHentaiFoundryTags( PageImport ):
def _InitManagementPanel( self ): self._management_panel = ClientGUIManagement.ManagementPanelImportWithQueueAdvancedHentaiFoundryTags( self._search_preview_split, self, self._page_key )
class PageImportNewgrounds( PageImport ):
def __init__( self, parent ):
PageImport.__init__( self, parent )
def _InitManagementPanel( self ): self._management_panel = ClientGUIManagement.ManagementPanelImportWithQueueAdvancedNewgrounds( self._search_preview_split, self, self._page_key )
class PageImportPixivArtist( PageImport ):
def __init__( self, parent ):

View File

@ -30,7 +30,7 @@ TEMP_DIR = BASE_DIR + os.path.sep + 'temp'
# Misc
NETWORK_VERSION = 9
SOFTWARE_VERSION = 69
SOFTWARE_VERSION = 70
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )
@ -269,6 +269,7 @@ header_and_mime = [
PREDICATE_TYPE_SYSTEM = 0
PREDICATE_TYPE_TAG = 1
PREDICATE_TYPE_NAMESPACE = 2
PREDICATE_TYPE_PARENT = 3
SITE_DOWNLOAD_TYPE_DEVIANT_ART = 0
SITE_DOWNLOAD_TYPE_GIPHY = 1
@ -276,6 +277,7 @@ SITE_DOWNLOAD_TYPE_PIXIV = 2
SITE_DOWNLOAD_TYPE_BOORU = 3
SITE_DOWNLOAD_TYPE_TUMBLR = 4
SITE_DOWNLOAD_TYPE_HENTAI_FOUNDRY = 5
SITE_DOWNLOAD_TYPE_NEWGROUNDS = 6
SYSTEM_PREDICATE_TYPE_EVERYTHING = 0
SYSTEM_PREDICATE_TYPE_INBOX = 1
@ -658,7 +660,7 @@ def ConvertTimestampToPrettyAge( timestamp ):
def ConvertTimestampToPrettyAgo( timestamp ):
if timestamp == 0: return 'unknown when'
if timestamp == 0: return 'unknown time'
age = int( time.time() ) - timestamp
@ -997,7 +999,7 @@ def SearchEntryMatchesTag( search_entry, tag ):
else: comparee = tag
if comparee.startswith( search_entry ): return True
if comparee.startswith( search_entry ) or ' ' + search_entry in comparee: return True
return False
@ -2061,6 +2063,16 @@ class Predicate():
if sibling is not None: base += u' (' + sibling + u')'
elif self._predicate_type == PREDICATE_TYPE_PARENT:
base = ' '
tag = self._value
base += tag
if self._count is not None: base += u' (' + ConvertIntToPrettyString( self._count ) + u')'
elif self._predicate_type == PREDICATE_TYPE_NAMESPACE:
( operator, namespace ) = self._value

View File

@ -34,6 +34,7 @@ def GetDownloader( site_download_type, *args ):
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_HENTAI_FOUNDRY: c = DownloaderHentaiFoundry
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_PIXIV: c = DownloaderPixiv
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_TUMBLR: c = DownloaderTumblr
elif site_download_type == HC.SITE_DOWNLOAD_TYPE_NEWGROUNDS: c = DownloaderNewgrounds
return c( *args )
@ -73,8 +74,6 @@ class Downloader():
self._num_pages_done = 0
example_url = self._GetNextGalleryPageURL()
def _EstablishSession( self, connection ): pass
@ -96,28 +95,35 @@ class Downloader():
return self._connections[ ( scheme, host, port ) ]
def _GetNextGalleryPageURLs( self ): return ( self._GetNextGalleryPageURL(), )
def GetAnotherPage( self ):
if self._we_are_done: return []
url = self._GetNextGalleryPageURL()
urls = self._GetNextGalleryPageURLs()
connection = self._GetConnection( url )
url_info = []
data = connection.geturl( url )
url_info = self._ParseGalleryPage( data, url )
for url in urls:
connection = self._GetConnection( url )
data = connection.geturl( url )
page_of_url_info = self._ParseGalleryPage( data, url )
# stop ourselves getting into an accidental infinite loop
url_info += [ info for info in page_of_url_info if info[0] not in self._all_urls_so_far ]
self._all_urls_so_far.update( [ info[0] for info in url_info ] )
# now url_info only contains new url info
self._num_pages_done += 1
# stop ourselves getting into an accidental infinite loop
url_info = [ info for info in url_info if info[0] not in self._all_urls_so_far ]
self._all_urls_so_far.update( [ info[0] for info in url_info ] )
# now url_info only contains new url info
return url_info
@ -648,6 +654,178 @@ class DownloaderHentaiFoundry( Downloader ):
connection.request( 'POST', '/site/filters', headers = headers, body = body )
class DownloaderNewgrounds( Downloader ):
def __init__( self, query ):
self._query = query
Downloader.__init__( self )
def _GetFileURLAndTags( self, url ):
connection = self._GetConnection( url )
html = connection.geturl( url )
return self._ParseImagePage( html, url )
def _GetNextGalleryPageURLs( self ):
artist = self._query
gallery_urls = []
gallery_urls.append( 'http://' + artist + '.newgrounds.com/games/' )
gallery_urls.append( 'http://' + artist + '.newgrounds.com/movies/' )
self._we_are_done = True
return gallery_urls
def _ParseGalleryPage( self, html, url_base ):
soup = bs4.BeautifulSoup( html )
fatcol = soup.find( 'div', class_ = 'fatcol' )
links = fatcol.find_all( 'a' )
urls_set = set()
result_urls = []
for link in links:
try:
url = link[ 'href' ]
if url not in urls_set:
if url.startswith( 'http://www.newgrounds.com/portal/view/' ):
urls_set.add( url )
result_urls.append( ( url, ) )
except: pass
return result_urls
def _ParseImagePage( self, html, url_base ):
soup = bs4.BeautifulSoup( html )
tags = []
author_links = soup.find( 'ul', class_ = 'authorlinks' )
if author_links is not None:
authors = set()
links = author_links.find_all( 'a' )
for link in links:
try:
href = link[ 'href' ] # http://warlord-of-noodles.newgrounds.com
creator = href.replace( 'http://', '' ).replace( '.newgrounds.com', '' )
tags.append( 'creator:' + creator )
except: pass
try:
title = soup.find( 'title' )
tags.append( 'title:' + title )
except: pass
all_links = soup.find_all( 'a' )
for link in all_links:
try:
href = link[ 'href' ]
if '/browse/tag/' in href: tags.append( link.string )
except: pass
#
try:
components = html.split( '"src"' )
# there is sometimes another bit of api flash earlier on that we don't want
# it is called http://uploads.ungrounded.net/apiassets/sandbox.swf
if len( components ) == 2: flash_url = components[1]
else: flash_url = components[2]
#"src": "http:\/\/flash.ngfiles.com\/video_player\/videoplayer.swf",
#"width": "100%",
#"src": "http:\/\/uploads.ungrounded.net\/593000\/593806_Kitty.swf",
#"width": "100%",
flash_url = flash_url.split( '"width"', 1 )[0]
flash_url = flash_url.split( '"' )[1]
flash_url = flash_url.replace( '\\/', '/' )
except: raise Exception( 'Could not find the swf file!' )
if flash_url == 'http://flash.ngfiles.com/video_player/videoplayer.swf': raise Exception( 'It was an mp4 movie, not a swf!' )
return ( flash_url, tags )
def GetFile( self, url ):
( file_url, tags ) = self._GetFileURLAndTags( url )
connection = self._GetConnection( file_url )
return connection.geturl( file_url )
def GetFileAndTags( self, url ):
( file_url, tags ) = self._GetFileURLAndTags( url )
connection = self._GetConnection( file_url )
file = connection.geturl( file_url )
return ( file, tags )
def GetTags( self, url ):
( file_url, tags ) = self._GetFileURLAndTags( url )
return tags
class DownloaderPixiv( Downloader ):
def __init__( self, query_type, query ):