Version 122

This commit is contained in:
Hydrus 2014-07-16 15:50:18 -05:00
parent 5fe43e355b
commit c0c66a2b24
19 changed files with 475 additions and 83 deletions

View File

@ -8,6 +8,21 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 122</h3></li>
<ul>
<li>fixed server upnp daemon to use correct port, I think</li>
<li>added client upnp daemon for local booru; it'll update upnp mappings as soon as they are changed</li>
<li>added bandwidth tracking for local booru; it'll even magically update before your eyes in review services</li>
<li>bandwidth limits are tested, so exceeding data usage will result in 403 errors until the next month</li>
<li>moved local booru css to an external file any user can edit</li>
<li>added 'open share in new page' button to local booru review services</li>
<li>fixed local booru display for video, flash, audio, pdf and miscellaneous</li>
<li>fixed local booru render resolution for all media</li>
<li>fixed local booru thumbnails for video, flash, audio, pdf and miscellaneous</li>
<li>added caching headers to static file requests to reduce browser-hydrus bandwidth</li>
<li>fixed local booru page text's newline formatting</li>
<li>added local booru unit tests</li>
</ul>
<li><h3>version 121</h3></li>
<ul>
<li>first version of local booru</li>

View File

@ -7,36 +7,34 @@
<body>
<div class="content">
<h3>local booru</h3>
<p>I have been working on a local booru, hosted from your client, to make it easier to share files with others over the internet. It is very new, but I will be adding more features to it. This help page will grow with it.</p>
<p>First of all, this is <b>hosted from your client</b>, which means other people will be connecting to your computer and fetching files you choose to share from your hard drive. If you close your client, the local booru will no longer work. And if you want people outside your home network (i.e. on the internet) to be able to see it, you will need to forward your client computer's booru port to your internet router.</p>
<p>The hydrus client has a simple booru to help you share your files with others over the internet.</p>
<p>First of all, this is <b>hosted from your client</b>, which means other people will be connecting to your computer and fetching files you choose to share from your hard drive. If you close your client, the local booru will no longer work.</p>
<h3>how to do it</h3>
<p>Right click some files you want to share and select <i>share->local booru</i>. This will throw up a small dialog, like so:</p>
<p><img src="local_booru_dialog.png" /></p>
<p>This lets you enter an optional <i>name</i>, which titles the share and helps you keep track of it, an optional <i>text</i>, which lets you say some words or html to the people you are sharing with, and an <i>expiry</i>, which lets you determine if and when the share will no longer work.</p>
<p>You can also copy either the internal or external link to your clipboard. The internal link (usually starting something like http://127.0.0.1:45866/) works inside your network and is great just for testing, while the external link (starting http://[your external ip address]:[external port]/) will work for anyone around the world, as long as your port is being forwarded correctly.</p>
<p>You can also copy either the internal or external link to your clipboard. The internal link (usually starting something like http://127.0.0.1:45866/) works inside your network and is great just for testing, while the external link (starting http://[your external ip address]:[external port]/) will work for anyone around the world, <b>as long as your booru's port is being forwarded correctly</b>.</p>
<p>If you use a dynamic-ip service like <a href="http://www.noip.com/">No-IP</a>, you can replace your external IP with your redirect hostname. You have to do it by hand right now, but I'll add a way to do it automatically in future.</p>
<p class="warning">Note that anyone with the external link will be able to see your share, so make sure you only share links with people you trust.</p>
<h3>forwarding your port</h3>
<p class="warning"><b>Hold the phone! I forgot to write the clientside UPnP Daemon to register the local booru's UPnP port! I'll add it next week, so for now, you probably want to add a custom entry via the <i>services->manage local upnp</i> dialog. Use (45866, TCP, 45866, hydrus client local booru, 604800). Send me an email if you can't figure it out.</b></p>
<p>Your home router acts as a barrier between the computers inside the network and the internet. Those inside can see out, but outsiders can only see what you tell the router to permit. Since you want to let people connect to your computer, you need to tell the router to forward all requests of a certain kind to your computer, and thus your client.</p>
<p>If you have never done this before, it can be a headache, especially doing it manually. Luckily, a technology called UPnP makes it a ton easier, and this is how your Skype or Bittorrent clients do it. I'm insane about privacy, though, so the hydrus network will not open any external ports without your permission. You need to go to <i>services->manage services</i> and uncheck a box:</p>
<p>If you have never done this before, it can be a headache, especially doing it manually. Luckily, a technology called UPnP makes it a ton easier, and this is how your Skype or Bittorrent clients do it automatically. Since I'm insane about privacy, the hydrus client will not open any external ports without your permission. You just need to go to <i>services->manage services</i> and uncheck a box:</p>
<p><img src="local_booru_upnp.png" /></p>
<p>Unless you know what you are doing and have a good reason to make them different, you might as well keep the internal and external ports the same. Once you have it set up, the client should make sure your router keeps that port open for your client. You should see the new mapping appear in your <i>services->manage local upnp</i> dialog, which lists all your router's current port mappings.</p>
<p>If you want to test that the port forward is set up correctly, just going to http://[external ip]:[external port]/ should give a little html just saying hello. Your ISP might not allow you to talk to yourself, though, so ask a friend to try if you are having trouble.</p>
<p>Unless you know what you are doing and have a good reason to make them different, you might as well keep the internal and external ports the same.</p>
<p>Once you have it set up, the client should make sure your router keeps that port open for your client. You should see the new mapping appear in your <i>services->manage local upnp</i> dialog, which lists all your router's current port mappings.</p>
<p>If you want to test that the port forward is set up correctly, going to http://[external ip]:[external port]/ should give a little html just saying hello. Your ISP might not allow you to talk to yourself, though, so ask a friend to try if you are having trouble.</p>
<p>If you still do not understand what is going on here, <a href="http://www.howtogeek.com/66214/how-to-forward-ports-on-your-router/">this</a> is a good article explaining everything.</p>
<p>If you do not like UPnP or your router does not support it, you can set the port forward up manually, but I encourage you to keep the internal and external port the same, because absent a 'upnp port' option, the 'copy external share link' button will use the internal port.</p>
<h3>so, what do you get?</h3>
<p>Since the booru is still prototype, the actual html layout is currently very simple:</p>
<p>The html layout is very simple:</p>
<hr />
<p><img src="local_booru_html.png" /></p>
<hr />
<p>It uses a very similar stylesheet to these help pages, although I've made the thumbnail grid look the way the client does it. At the moment, the thumbnails are the gigantic max size of 200x200, but I will add an option to display the resized versions. Also, flash and audio files will 404 on the thumbnail for the moment.</p>
<p>Clicking a thumbnail leads to a barebones image page that I will flush out in future.</p>
<p>It uses a very similar stylesheet to these help pages. If you would like to change the style, feel free to have a look at the very simple html and then edit install_dir/static/local_booru_style.css. The thumbnails will be the same size as in your client.</p>
<h3>editing an existing share</h3>
<p>You can review all your shares on <i>services->review services</i>, under <i>local->booru</i>. You can copy the links again, change the title/text/expiration, and delete any shares you don't want any more.</p>
<h3>future plans</h3>
<p>Now I have the simplest possible booru working, there is a lot I can add. I would like to make the thumbnails resizeable, and add sorting controls, allow for custom css, add a normal tag display, make those tags searchable, and generally AJAX-ify the entire thing to offload CPU time to the client's web browser.</p>
<p>Also, the bandwidth tracking feature does not work yet, so I'll get that going.</p>
<p>Now I have a simple booru working, I would like to add sorting controls, tag display, make those tags searchable, and generally AJAX-ify the entire thing to reduce bandwidth and offload CPU time to the web browser.</p>
</div>
</body>
</html>

Binary file not shown.

Before

Width:  |  Height:  |  Size: 227 KiB

After

Width:  |  Height:  |  Size: 61 KiB

View File

@ -1097,6 +1097,17 @@ class LocalBooruCache( object ):
self._RefreshShares()
HC.pubsub.sub( self, 'RefreshShares', 'refresh_local_booru_shares' )
HC.pubsub.sub( self, 'RefreshShares', 'restart_booru' )
def _CheckDataUsage( self ):
info = self._local_booru_service.GetInfo()
max_monthly_data = info[ 'max_monthly_data' ]
used_monthly_data = info[ 'used_monthly_data' ]
if max_monthly_data is not None and used_monthly_data > max_monthly_data: raise HydrusExceptions.ForbiddenException( 'This booru has used all its monthly data. Please try again next month.' )
def _CheckFileAuthorised( self, share_key, hash ):
@ -1110,6 +1121,8 @@ class LocalBooruCache( object ):
def _CheckShareAuthorised( self, share_key ):
self._CheckDataUsage()
info = self._GetInfo( share_key )
timeout = info[ 'timeout' ]
@ -1130,6 +1143,14 @@ class LocalBooruCache( object ):
info[ 'hashes_set' ] = set( hashes )
media_results = HC.app.ReadDaemon( 'media_results', HC.LOCAL_FILE_SERVICE_IDENTIFIER, hashes )
info[ 'media_results' ] = media_results
hashes_to_media_results = { media_result.GetHash() : media_result for media_result in media_results }
info[ 'hashes_to_media_results' ] = hashes_to_media_results
self._keys_to_infos[ share_key ] = info
@ -1138,6 +1159,8 @@ class LocalBooruCache( object ):
def _RefreshShares( self ):
self._local_booru_service = HC.app.Read( 'service', HC.LOCAL_BOORU_SERVICE_IDENTIFIER )
self._keys_to_infos = {}
share_keys = HC.app.Read( 'local_booru_share_keys' )
@ -1159,17 +1182,28 @@ class LocalBooruCache( object ):
with self._lock:
self._CheckShareAuthorised( share_key )
info = self._GetInfo( share_key )
name = info[ 'name' ]
text = info[ 'text' ]
timeout = info[ 'timeout' ]
hashes = info[ 'hashes' ]
media_results = info[ 'media_results' ]
# eventually move to a set of media results that has tags from ~whatever~
# the media_results should be generated on demand and cached with a timeout
return ( name, text, timeout, media_results )
return ( name, text, timeout, hashes )
def GetMediaResult( self, share_key, hash ):
with self._lock:
info = self._GetInfo( share_key )
media_result = info[ 'hashes_to_media_results' ][ hash ]
return media_result
@ -1177,15 +1211,16 @@ class LocalBooruCache( object ):
with self._lock:
self._CheckFileAuthorised( share_key, hash )
info = self._GetInfo( share_key )
name = info[ 'name' ]
text = info[ 'text' ]
timeout = info[ 'timeout' ]
media_result = info[ 'hashes_to_media_results' ][ hash ]
# eventually move to returning resolution and so on so we can show tags and whatever, doing a proper page
return ( name, text, timeout )
return ( name, text, timeout, media_result )
@ -2385,7 +2420,14 @@ class Service( HC.HydrusYAMLBase ):
num_bytes = row
self._info[ 'account' ].RequestMade( num_bytes )
service_type = self._service_identifier.GetType()
if service_type == HC.LOCAL_BOORU:
self._info[ 'used_monthly_data' ] += num_bytes
self._info[ 'used_monthly_requests' ] += 1
else: self._info[ 'account' ].RequestMade( num_bytes )
elif action == HC.SERVICE_UPDATE_NEXT_DOWNLOAD_TIMESTAMP:
@ -2536,11 +2578,11 @@ class ThumbnailCache():
thumbnail = HydrusFileHandling.GenerateThumbnail( path, HC.options[ 'thumbnail_dimensions' ] )
temp_path = HC.GetTempPath()
resized_path = HC.STATIC_DIR + os.path.sep + name + '_resized.png'
with open( temp_path, 'wb' ) as f: f.write( thumbnail )
with open( resized_path, 'wb' ) as f: f.write( thumbnail )
hydrus_bitmap = HydrusImageHandling.GenerateHydrusBitmap( temp_path )
hydrus_bitmap = HydrusImageHandling.GenerateHydrusBitmap( resized_path )
self._special_thumbs[ name ] = hydrus_bitmap

View File

@ -10,6 +10,7 @@ import HydrusExceptions
import HydrusFileHandling
import HydrusImageHandling
import HydrusMessageHandling
import HydrusNATPunch
import HydrusServer
import HydrusTags
import HydrusThreading
@ -1325,9 +1326,15 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
if service_type == HC.LOCAL_BOORU:
current_time_struct = time.gmtime()
( current_year, current_month ) = ( current_time_struct.tm_year, current_time_struct.tm_mon )
if 'used_monthly_data' not in info: info[ 'used_monthly_data' ] = 0
if 'max_monthly_data' not in info: info[ 'max_monthly_data' ] = None
if 'port' not in info: info[ 'port' ] = 45866
if 'used_monthly_requests' not in info: info[ 'used_monthly_requests' ] = 0
if 'current_data_month' not in info: info[ 'current_data_month' ] = ( current_year, current_month )
if 'port' not in info: info[ 'port' ] = HC.DEFAULT_LOCAL_BOORU_PORT
if 'upnp' not in info: info[ 'upnp' ] = None
@ -3652,10 +3659,13 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
do_new_permissions = False
requests_made = []
hydrus_requests_made = []
local_booru_requests_made = []
for ( service_identifier, service_updates ) in service_identifiers_to_service_updates.items():
service_type = service_identifier.GetType()
try: service_id = self._GetServiceId( c, service_identifier )
except: continue
@ -3683,7 +3693,8 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
num_bytes = row
requests_made.append( ( service_id, num_bytes ) )
if service_type == HC.LOCAL_BOORU: local_booru_requests_made.append( num_bytes )
else: hydrus_requests_made.append( ( service_id, num_bytes ) )
elif action == HC.SERVICE_UPDATE_NEWS:
@ -3735,7 +3746,7 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
self.pub_service_updates_after_commit( service_identifiers_to_service_updates )
for ( service_id, nums_bytes ) in HC.BuildKeyToListDict( requests_made ).items():
for ( service_id, nums_bytes ) in HC.BuildKeyToListDict( hydrus_requests_made ).items():
( info, ) = c.execute( 'SELECT info FROM services WHERE service_id = ?;', ( service_id, ) ).fetchone()
@ -3746,6 +3757,32 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
c.execute( 'UPDATE services SET info = ? WHERE service_id = ?;', ( info, service_id ) )
if len( local_booru_requests_made ) > 0:
service_id = self._GetServiceId( c, HC.LOCAL_BOORU_SERVICE_IDENTIFIER )
( info, ) = c.execute( 'SELECT info FROM services WHERE service_id = ?;', ( service_id, ) ).fetchone()
current_time_struct = time.gmtime()
( current_year, current_month ) = ( current_time_struct.tm_year, current_time_struct.tm_mon )
( booru_year, booru_month ) = info[ 'current_data_month' ]
if current_year != booru_year or current_month != booru_month:
info[ 'used_monthly_data' ] = 0
info[ 'used_monthly_requests' ] = 0
info[ 'current_data_month' ] = ( current_year, current_month )
info[ 'used_monthly_data' ] += sum( local_booru_requests_made )
info[ 'used_monthly_requests' ] += len( local_booru_requests_made )
c.execute( 'UPDATE services SET info = ? WHERE service_id = ?;', ( info, service_id ) )
if do_new_permissions: self.pub_after_commit( 'notify_new_permissions' )
@ -4426,7 +4463,11 @@ class ServiceDB( FileDB, MessageDB, TagDB, RatingDB ):
self._UpdateServiceInfo( c, service_id, update )
if service_type == HC.LOCAL_BOORU: HC.pubsub.pub( 'restart_booru' )
if service_type == HC.LOCAL_BOORU:
HC.pubsub.pub( 'restart_booru' )
HC.pubsub.pub( 'notify_new_upnp_mappings' )
@ -4637,8 +4678,6 @@ class DB( ServiceDB ):
c.execute( 'CREATE TABLE autocomplete_tags_cache ( file_service_id INTEGER REFERENCES services ( service_id ) ON DELETE CASCADE, tag_service_id INTEGER REFERENCES services ( service_id ) ON DELETE CASCADE, namespace_id INTEGER, tag_id INTEGER, current_count INTEGER, pending_count INTEGER, PRIMARY KEY ( file_service_id, tag_service_id, namespace_id, tag_id ) );' )
c.execute( 'CREATE INDEX autocomplete_tags_cache_tag_service_id_namespace_id_tag_id_index ON autocomplete_tags_cache ( tag_service_id, namespace_id, tag_id );' )
c.execute( 'CREATE TABLE booru_shares ( service_id INTEGER REFERENCES services ( service_id ) ON DELETE CASCADE, share_key BLOB_BYTES, share TEXT_YAML, expiry INTEGER, used_monthly_data INTEGER, max_monthly_data INTEGER, ip_restriction TEXT, notes TEXT, PRIMARY KEY( service_id, share_key ) );' )
c.execute( 'CREATE TABLE contacts ( contact_id INTEGER PRIMARY KEY, contact_key BLOB_BYTES, public_key TEXT, name TEXT, host TEXT, port INTEGER );' )
c.execute( 'CREATE UNIQUE INDEX contacts_contact_key_index ON contacts ( contact_key );' )
c.execute( 'CREATE UNIQUE INDEX contacts_name_index ON contacts ( name );' )
@ -5003,6 +5042,24 @@ class DB( ServiceDB ):
if version == 121:
c.execute( 'DROP TABLE booru_shares;' )
service_id = self._GetServiceId( c, HC.LOCAL_BOORU_SERVICE_IDENTIFIER )
( info, ) = c.execute( 'SELECT info FROM services WHERE service_id = ?;', ( service_id, ) ).fetchone()
current_time_struct = time.gmtime()
( current_year, current_month ) = ( current_time_struct.tm_year, current_time_struct.tm_mon )
info[ 'used_monthly_requests' ] = 0
info[ 'current_data_month' ] = ( current_year, current_month )
c.execute( 'UPDATE services SET info = ? WHERE service_id = ?;', ( info, service_id ) )
c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
HC.is_db_updated = True
@ -7248,6 +7305,7 @@ class DB( ServiceDB ):
HydrusThreading.DAEMONWorker( 'SynchroniseAccounts', DAEMONSynchroniseAccounts, ( 'notify_new_services', 'permissions_are_stale' ) )
HydrusThreading.DAEMONWorker( 'SynchroniseRepositories', DAEMONSynchroniseRepositories, ( 'notify_restart_repo_sync_daemon', 'notify_new_permissions' ) )
HydrusThreading.DAEMONWorker( 'SynchroniseSubscriptions', DAEMONSynchroniseSubscriptions, ( 'notify_restart_subs_sync_daemon', 'notify_new_subscriptions' ) )
HydrusThreading.DAEMONWorker( 'UPnP', DAEMONUPnP, ( 'notify_new_upnp_mappings', ) )
HydrusThreading.DAEMONQueue( 'FlushRepositoryUpdates', DAEMONFlushServiceUpdates, 'service_updates_delayed', period = 5 )
@ -8420,4 +8478,55 @@ def DAEMONSynchroniseSubscriptions():
time.sleep( 3 )
def DAEMONUPnP():
local_ip = HydrusNATPunch.GetLocalIP()
current_mappings = HydrusNATPunch.GetUPnPMappings()
our_mappings = { ( internal_client, internal_port ) : external_port for ( description, internal_client, internal_port, external_ip_address, external_port, protocol, enabled ) in current_mappings }
services = HC.app.ReadDaemon( 'services', ( HC.LOCAL_BOORU, ) )
for service in services:
info = service.GetInfo()
internal_port = info[ 'port' ]
upnp = info[ 'upnp' ]
if ( local_ip, internal_port ) in our_mappings:
current_external_port = our_mappings[ ( local_ip, internal_port ) ]
if upnp is None or current_external_port != upnp: HydrusNATPunch.RemoveUPnPMapping( current_external_port, 'TCP' )
for service in services:
info = service.GetInfo()
internal_port = info[ 'port' ]
upnp = info[ 'upnp' ]
if upnp is not None:
if ( local_ip, internal_port ) not in our_mappings:
service_identifier = service.GetServiceIdentifier()
external_port = upnp
protocol = 'TCP'
description = HC.service_string_lookup[ service_identifier.GetType() ] + ' at ' + local_ip + ':' + str( internal_port )
duration = 3600
HydrusNATPunch.AddUPnPMapping( local_ip, internal_port, external_port, protocol, description, duration = duration )

View File

@ -2319,6 +2319,9 @@ class FrameReviewServices( ClientGUICommon.Frame ):
self._booru_shares = ClientGUICommon.SaneListCtrl( self._booru_shares_panel, -1, [ ( 'title', 110 ), ( 'text', -1 ), ( 'expires', 170 ), ( 'num files', 70 ) ] )
self._booru_open_search = wx.Button( self._booru_shares_panel, label = 'open share in new page' )
self._booru_open_search.Bind( wx.EVT_BUTTON, self.EventBooruOpenSearch )
self._copy_internal_share_link = wx.Button( self._booru_shares_panel, label = 'copy internal share link' )
self._copy_internal_share_link.Bind( wx.EVT_BUTTON, self.EventCopyInternalShareURL )
@ -2424,6 +2427,7 @@ class FrameReviewServices( ClientGUICommon.Frame ):
self._booru_shares_panel.AddF( self._booru_shares, FLAGS_EXPAND_BOTH_WAYS )
b_box = wx.BoxSizer( wx.HORIZONTAL )
b_box.AddF( self._booru_open_search, FLAGS_MIXED )
b_box.AddF( self._copy_internal_share_link, FLAGS_MIXED )
b_box.AddF( self._copy_external_share_link, FLAGS_MIXED )
b_box.AddF( self._booru_edit, FLAGS_MIXED )
@ -2501,12 +2505,16 @@ class FrameReviewServices( ClientGUICommon.Frame ):
max_monthly_data = info[ 'max_monthly_data' ]
used_monthly_data = info[ 'used_monthly_data' ]
used_monthly_requests = info[ 'used_monthly_requests' ]
if used_monthly_requests == 0: monthly_requests_text = ''
else: monthly_requests_text = ' in ' + HC.ConvertIntToPrettyString( used_monthly_requests ) + ' requests'
if max_monthly_data is None:
self._bytes.Hide()
self._bytes_text.SetLabel( 'used ' + HC.ConvertIntToBytes( used_monthly_data ) + ' this month' )
self._bytes_text.SetLabel( 'used ' + HC.ConvertIntToBytes( used_monthly_data ) + monthly_requests_text + ' this month' )
else:
@ -2515,7 +2523,7 @@ class FrameReviewServices( ClientGUICommon.Frame ):
self._bytes.SetRange( max_monthly_data )
self._bytes.SetValue( used_monthly_data )
self._bytes_text.SetLabel( 'used ' + HC.ConvertIntToBytes( used_monthly_data ) + '/' + HC.ConvertIntToBytes( max_monthly_data ) + ' this month' )
self._bytes_text.SetLabel( 'used ' + HC.ConvertIntToBytes( used_monthly_data ) + '/' + HC.ConvertIntToBytes( max_monthly_data ) + monthly_requests_text + ' this month' )
@ -2743,6 +2751,16 @@ class FrameReviewServices( ClientGUICommon.Frame ):
for ( share_key, info ) in writes: HC.app.Write( 'local_booru_share', share_key, info )
def EventBooruOpenSearch( self, event ):
for ( name, text, timeout, ( num_hashes, hashes, share_key ) ) in self._booru_shares.GetSelectedClientData():
media_results = HC.app.Read( 'media_results', HC.LOCAL_FILE_SERVICE_IDENTIFIER, hashes )
HC.pubsub.pub( 'new_page_query', HC.LOCAL_FILE_SERVICE_IDENTIFIER, initial_media_results = media_results )
def EventCopyExternalShareURL( self, event ):
shares = self._booru_shares.GetSelectedClientData()

View File

@ -64,7 +64,7 @@ options = {}
# Misc
NETWORK_VERSION = 13
SOFTWARE_VERSION = 121
SOFTWARE_VERSION = 122
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )
@ -572,6 +572,7 @@ REQUESTS_TO_PERMISSIONS = { ( service_type, request_type, request ) : permission
# default options
DEFAULT_LOCAL_FILE_PORT = 45865
DEFAULT_LOCAL_BOORU_PORT = 45866
DEFAULT_SERVER_ADMIN_PORT = 45870
DEFAULT_SERVICE_PORT = 45871

View File

@ -15,12 +15,15 @@ if HC.PLATFORM_LINUX: upnpc_path = '"' + HC.BIN_DIR + os.path.sep + 'upnpc_linux
elif HC.PLATFORM_OSX: upnpc_path = '"' + HC.BIN_DIR + os.path.sep + 'upnpc_osx"'
elif HC.PLATFORM_WINDOWS: upnpc_path = '"' + HC.BIN_DIR + os.path.sep + 'upnpc_win32.exe"'
external_ip = None
external_ip_time = 0
EXTERNAL_IP = {}
EXTERNAL_IP[ 'ip' ] = None
EXTERNAL_IP[ 'time' ] = 0
def GetExternalIP():
if HC.GetNow() - external_ip_time > 3600 * 24:
if HC.GetNow() - EXTERNAL_IP[ 'time' ] > 3600 * 24:
EXTERNAL_IP[ 'time' ] = HC.GetNow()
command = upnpc_path + ' -l'
@ -41,13 +44,13 @@ def GetExternalIP():
'''ExternalIPAddress = ip'''
( gumpf, external_ip ) = lines[ i - 1 ].split( ' = ' )
( gumpf, EXTERNAL_IP[ 'ip' ] ) = lines[ i - 1 ].split( ' = ' )
except: raise Exception( 'Problem while trying to fetch External IP.' )
return external_ip
return EXTERNAL_IP[ 'ip' ]
def GetLocalIP(): return socket.gethostbyname( socket.gethostname() )

View File

@ -312,6 +312,7 @@ class HydrusServiceBooru( HydrusService ):
root.putChild( 'page', HydrusServerResources.HydrusResourceCommandBooruPage( self._service_identifier, REMOTE_DOMAIN ) )
root.putChild( 'file', HydrusServerResources.HydrusResourceCommandBooruFile( self._service_identifier, REMOTE_DOMAIN ) )
root.putChild( 'thumbnail', HydrusServerResources.HydrusResourceCommandBooruThumbnail( self._service_identifier, REMOTE_DOMAIN ) )
root.putChild( 'style.css', HydrusServerResources.local_booru_css )
return root

View File

@ -16,6 +16,7 @@ import random
import ServerConstants as SC
import SocketServer
import threading
import time
import traceback
import urllib
import wx
@ -266,7 +267,8 @@ def ParseFileArguments( path ):
return args
hydrus_favicon = FileResource( HC.STATIC_DIR + os.path.sep + 'hydrus.ico', defaultType = HC.IMAGE_ICON )
hydrus_favicon = FileResource( HC.STATIC_DIR + os.path.sep + 'hydrus.ico', defaultType = 'image/x-icon' )
local_booru_css = FileResource( HC.STATIC_DIR + os.path.sep + 'local_booru_style.css', defaultType = 'text/css' )
class HydrusDomain( object ):
@ -466,6 +468,9 @@ class HydrusResourceCommand( Resource ):
request.setHeader( 'Content-Type', content_type )
request.setHeader( 'Content-Length', str( content_length ) )
request.setHeader( 'Expires', time.strftime( '%a, %d %b %Y %H:%M:%S GMT', time.gmtime( time.time() + 86400 * 365 ) ) )
request.setHeader( 'Cache-Control', str( 86400 * 365 ) )
fileObject = open( path, 'rb' )
producer = NoRangeStaticProducer( request, fileObject )
@ -627,7 +632,7 @@ class HydrusResourceCommand( Resource ):
return access_key
def _recordDataUsage( self, request ): return request
def _recordDataUsage( self, request ): pass
def _threadDoGETJob( self, request ): raise HydrusExceptions.NotFoundException( 'This service does not support that request!' )
@ -719,20 +724,37 @@ class HydrusResourceCommandAccessKeyVerification( HydrusResourceCommand ):
class HydrusResourceCommandBooru( HydrusResourceCommand ):
RECORD_GET_DATA_USAGE = False
RECORD_POST_DATA_USAGE = False
def _recordDataUsage( self, request ):
p1 = request.method == 'GET' and self.RECORD_GET_DATA_USAGE
p2 = request.method == 'POST' and self.RECORD_POST_DATA_USAGE
if p1 or p2:
num_bytes = request.hydrus_request_data_usage
service_identifier = HC.LOCAL_BOORU_SERVICE_IDENTIFIER # this is important, as self._service_identifier is the server identifier
HC.pubsub.pub( 'service_updates_delayed', { service_identifier : [ HC.ServiceUpdate( HC.SERVICE_UPDATE_REQUEST_MADE, num_bytes ) ] } )
def _callbackCheckRestrictions( self, request ):
self._checkUserAgent( request )
self._domain.CheckValid( request.getClientIP() )
# extract booru key, hence fetch booru cache thing and assign it to this request
# or fail with a 404
return request
class HydrusResourceCommandBooruGallery( HydrusResourceCommandBooru ):
RECORD_GET_DATA_USAGE = True
def _threadDoGETJob( self, request ):
# in future, make this a standard frame with a search key that'll load xml or yaml AJAX stuff
@ -744,7 +766,7 @@ class HydrusResourceCommandBooruGallery( HydrusResourceCommandBooru ):
local_booru_manager.CheckShareAuthorised( share_key )
( name, text, timeout, hashes ) = local_booru_manager.GetGalleryInfo( share_key )
( name, text, timeout, media_results ) = local_booru_manager.GetGalleryInfo( share_key )
body = '''<html>
<head>'''
@ -755,27 +777,26 @@ class HydrusResourceCommandBooruGallery( HydrusResourceCommandBooru ):
<title>''' + name + '''</title>'''
body += '''
<link href="hydrus.ico" rel="shortcut icon" />
<link href="style.css" rel="stylesheet" type="text/css" />'''
( thumbnail_width, thumbnail_height ) = HC.options[ 'thumbnail_dimensions' ]
body += '''
<style>
.thumbnail_container { width: ''' + str( thumbnail_width ) + '''px; height: ''' + str( thumbnail_height ) + '''px; }
</style>'''
body += '''
</head>
<style>
body { font-family: "Calibri", Arial, sans-serif; color: rgb( 85, 85, 85 ); line-height: 1.5; }
a { color: #222; text-decoration: none; font-weight: bold; }
h3 { color: #222; }
a:hover { color: #555 }
.footer { text-align: center; font-size: 80% }
.media { clear: both; }
.name { font-weight: bold; font-size: 150%; }
.thumbnail { margin: 1px; border: 1px rgb( 223, 227, 230 ) solid; display: inline-block; }
.thumbnail_container { text-align: center; width: 200px; height: 200px; display: table-cell; vertical-align: middle; }
.thumbnail_container img { display: block; margin-left: auto; margin-right: auto; }
.timeout { font-size: 80%; float: right; margin: 2px }
</style>
<body>'''
body += '''
<span class="timeout">This share ''' + HC.ConvertTimestampToPrettyExpiry( timeout ) + '''.</span>'''
<div class="timeout">This share ''' + HC.ConvertTimestampToPrettyExpiry( timeout ) + '''.</div>'''
if name != '': body += '''
<div class="name">''' + name + '''</div>'''
<h3>''' + name + '''</h3>'''
if text != '':
@ -788,7 +809,12 @@ a:hover { color: #555 }
body+= '''
<div class="media">'''
for hash in hashes:
for media_result in media_results:
hash = media_result.GetHash()
mime = media_result.GetMime()
# if mime in flash or pdf or whatever, get other thumbnail
body += '''
<span class="thumbnail">
@ -813,6 +839,8 @@ a:hover { color: #555 }
class HydrusResourceCommandBooruFile( HydrusResourceCommandBooru ):
RECORD_GET_DATA_USAGE = True
def _threadDoGETJob( self, request ):
share_key = request.hydrus_args[ 'share_key' ]
@ -831,6 +859,8 @@ class HydrusResourceCommandBooruFile( HydrusResourceCommandBooru ):
class HydrusResourceCommandBooruPage( HydrusResourceCommandBooru ):
RECORD_GET_DATA_USAGE = True
def _threadDoGETJob( self, request ):
share_key = request.hydrus_args[ 'share_key' ]
@ -840,7 +870,7 @@ class HydrusResourceCommandBooruPage( HydrusResourceCommandBooru ):
local_booru_manager.CheckFileAuthorised( share_key, hash )
( name, text, timeout ) = local_booru_manager.GetPageInfo( share_key, hash )
( name, text, timeout, media_result ) = local_booru_manager.GetPageInfo( share_key, hash )
body = '''<html>
<head>'''
@ -851,27 +881,19 @@ class HydrusResourceCommandBooruPage( HydrusResourceCommandBooru ):
<title>''' + name + '''</title>'''
body += '''
<link href="hydrus.ico" rel="shortcut icon" />
<link href="style.css" rel="stylesheet" type="text/css" />'''
body += '''
</head>
<style>
body { font-family: "Calibri", Arial, sans-serif; color: rgb( 85, 85, 85 ); line-height: 1.5; }
a { color: #222; text-decoration: none; font-weight: bold; }
h3 { color: #222; }
a:hover { color: #555 }
.footer { text-align: center; font-size: 80% }
.media { clear: both; }
.name { font-weight: bold; font-size: 150%; }
.thumbnail { margin: 1px; border: 1px rgb( 223, 227, 230 ) solid; display: inline-block; }
.thumbnail_container { text-align: center; width: 200px; height: 200px; display: table-cell; vertical-align: middle; }
.thumbnail_container img { display: block; margin-left: auto; margin-right: auto; }
.timeout { font-size: 80%; float: right; margin: 2px }
</style>
<body>'''
body += '''
<span class="timeout">This share ''' + HC.ConvertTimestampToPrettyExpiry( timeout ) + '''.</span>'''
<div class="timeout">This share ''' + HC.ConvertTimestampToPrettyExpiry( timeout ) + '''.</div>'''
if name != '': body += '''
<div class="name">''' + name + '''</div>'''
<h3>''' + name + '''</h3>'''
if text != '':
@ -879,11 +901,43 @@ a:hover { color: #555 }
<p>'''
body += '''
<p>''' + text.replace( os.linesep, newline ) + '''</p>'''
<p>''' + text.replace( os.linesep, newline ).replace( '\n', newline ) + '''</p>'''
body+= '''
<div class="media">
<img src="file?share_key=''' + share_key.encode( 'hex' ) + '''&hash=''' + hash.encode( 'hex' ) + '''" />
<div class="media">'''
mime = media_result.GetMime()
if mime in HC.IMAGES:
( width, height ) = media_result.GetResolution()
body += '''
<img width="''' + str( width ) + '''" height="''' + str( height ) + '''" src="file?share_key=''' + share_key.encode( 'hex' ) + '''&hash=''' + hash.encode( 'hex' ) + '''" />'''
elif mime in HC.VIDEO:
( width, height ) = media_result.GetResolution()
body += '''
<video width="''' + str( width ) + '''" height="''' + str( height ) + '''" controls="" loop="" autoplay="" src="file?share_key=''' + share_key.encode( 'hex' ) + '''&hash=''' + hash.encode( 'hex' ) + '''" />
<p><a href="file?share_key=''' + share_key.encode( 'hex' ) + '''&hash=''' + hash.encode( 'hex' ) + '''">link to ''' + HC.mime_string_lookup[ mime ] + ''' file</a></p>'''
elif mime == HC.APPLICATION_FLASH:
( width, height ) = media_result.GetResolution()
body += '''
<embed width="''' + str( width ) + '''" height="''' + str( height ) + '''" src="file?share_key=''' + share_key.encode( 'hex' ) + '''&hash=''' + hash.encode( 'hex' ) + '''" />
<p><a href="file?share_key=''' + share_key.encode( 'hex' ) + '''&hash=''' + hash.encode( 'hex' ) + '''">link to ''' + HC.mime_string_lookup[ mime ] + ''' file</a></p>'''
else:
body += '''
<p><a href="file?share_key=''' + share_key.encode( 'hex' ) + '''&hash=''' + hash.encode( 'hex' ) + '''">link to ''' + HC.mime_string_lookup[ mime ] + ''' file</a></p>'''
body += '''
</div>
<div class="footer"><a href="http://hydrusnetwork.github.io/hydrus/">hydrus network</a></div>
</body>
@ -896,6 +950,8 @@ a:hover { color: #555 }
class HydrusResourceCommandBooruThumbnail( HydrusResourceCommandBooru ):
RECORD_GET_DATA_USAGE = True
def _threadDoGETJob( self, request ):
share_key = request.hydrus_args[ 'share_key' ]
@ -905,7 +961,16 @@ class HydrusResourceCommandBooruThumbnail( HydrusResourceCommandBooru ):
local_booru_manager.CheckFileAuthorised( share_key, hash )
path = CC.GetThumbnailPath( hash )
media_result = local_booru_manager.GetMediaResult( share_key, hash )
mime = media_result.GetMime()
if mime in HC.MIMES_WITH_THUMBNAILS: path = CC.GetThumbnailPath( hash, full_size = False )
elif mime in HC.AUDIO: path = HC.STATIC_DIR + os.path.sep + 'audio_resized.png'
elif mime in HC.VIDEO: path = HC.STATIC_DIR + os.path.sep + 'video_resized.png'
elif mime == HC.APPLICATION_FLASH: path = HC.STATIC_DIR + os.path.sep + 'flash_resized.png'
elif mime == HC.APPLICATION_PDF: path = HC.STATIC_DIR + os.path.sep + 'pdf_resized.png'
else: path = HC.STATIC_DIR + os.path.sep + 'hydrus_resized.png'
response_context = HC.ResponseContext( 200, path = path )

View File

@ -2906,7 +2906,7 @@ def DAEMONUPnP():
if ( local_ip, internal_port ) not in our_mappings:
external_port = our_mappings[ ( local_ip, internal_port ) ]
external_port = upnp
protocol = 'TCP'

View File

@ -26,6 +26,7 @@ class TestServer( unittest.TestCase ):
services = []
self._local_service_identifier = HC.ServerServiceIdentifier( 'local file', HC.LOCAL_FILE )
self._booru_service_identifier = HC.ServerServiceIdentifier( 'local booru', HC.LOCAL_BOORU )
self._file_service_identifier = HC.ServerServiceIdentifier( 'file service', HC.FILE_REPOSITORY )
self._tag_service_identifier = HC.ServerServiceIdentifier( 'tag service', HC.TAG_REPOSITORY )
@ -46,6 +47,7 @@ class TestServer( unittest.TestCase ):
reactor.listenTCP( HC.DEFAULT_SERVER_ADMIN_PORT, HydrusServer.HydrusServiceAdmin( HC.SERVER_ADMIN_IDENTIFIER, 'hello' ) )
reactor.listenTCP( HC.DEFAULT_LOCAL_FILE_PORT, HydrusServer.HydrusServiceLocal( self._local_service_identifier, 'hello' ) )
reactor.listenTCP( HC.DEFAULT_LOCAL_BOORU_PORT, HydrusServer.HydrusServiceBooru( self._booru_service_identifier, 'hello' ) )
reactor.listenTCP( HC.DEFAULT_SERVICE_PORT, HydrusServer.HydrusServiceRepositoryFile( self._file_service_identifier, 'hello' ) )
reactor.listenTCP( HC.DEFAULT_SERVICE_PORT + 1, HydrusServer.HydrusServiceRepositoryTag( self._tag_service_identifier, 'hello' ) )
@ -193,6 +195,101 @@ class TestServer( unittest.TestCase ):
except: pass
def _test_local_booru( self, host, port ):
connection = httplib.HTTPConnection( host, port, timeout = 10 )
#
with open( HC.STATIC_DIR + os.path.sep + 'local_booru_style.css', 'rb' ) as f: css = f.read()
connection.request( 'GET', '/style.css' )
response = connection.getresponse()
data = response.read()
self.assertEqual( data, css )
#
share_key = os.urandom( 32 )
hashes = [ os.urandom( 32 ) for i in range( 5 ) ]
with open( CC.GetExpectedFilePath( hashes[0], HC.IMAGE_JPEG ), 'wb' ) as f: f.write( 'file' )
with open( CC.GetExpectedThumbnailPath( hashes[0], False ), 'wb' ) as f: f.write( 'thumbnail' )
local_booru_manager = HC.app.GetManager( 'local_booru' )
#
self._test_local_booru_requests( connection, share_key, hashes[0], 404 )
#
info = {}
info[ 'name' ] = 'name'
info[ 'text' ] = 'text'
info[ 'timeout' ] = 0
info[ 'hashes' ] = hashes
# hash, inbox, size, mime, timestamp, width, height, duration, num_frames, num_words, tags_manager, file_service_identifiers_cdpp, local_ratings, remote_ratings
media_results = [ CC.MediaResult( ( hash, True, 500, HC.IMAGE_JPEG, 0, 640, 480, None, None, None, None, None, None, None ) ) for hash in hashes ]
HC.app.SetRead( 'local_booru_share_keys', [ share_key ] )
HC.app.SetRead( 'local_booru_share', info )
HC.app.SetRead( 'media_results', media_results )
local_booru_manager.RefreshShares()
#
self._test_local_booru_requests( connection, share_key, hashes[0], 403 )
#
info[ 'timeout' ] = None
HC.app.SetRead( 'local_booru_share', info )
local_booru_manager.RefreshShares()
#
self._test_local_booru_requests( connection, share_key, hashes[0], 200 )
#
HC.app.SetRead( 'local_booru_share_keys', [] )
local_booru_manager.RefreshShares()
#
self._test_local_booru_requests( connection, share_key, hashes[0], 404 )
def _test_local_booru_requests( self, connection, share_key, hash, expected_result ):
requests = []
requests.append( '/gallery?share_key=' + share_key.encode( 'hex' ) )
requests.append( '/page?share_key=' + share_key.encode( 'hex' ) + '&hash=' + hash.encode( 'hex' ) )
requests.append( '/file?share_key=' + share_key.encode( 'hex' ) + '&hash=' + hash.encode( 'hex' ) )
requests.append( '/thumbnail?share_key=' + share_key.encode( 'hex' ) + '&hash=' + hash.encode( 'hex' ) )
for request in requests:
connection.request( 'GET', request )
response = connection.getresponse()
data = response.read()
self.assertEqual( response.status, expected_result )
def _test_repo( self, host, port, service_type ):
service_identifier = HC.ClientServiceIdentifier( os.urandom( 32 ), service_type, 'service' )
@ -483,6 +580,15 @@ class TestServer( unittest.TestCase ):
self._test_server_admin( host, port )
def test_local_booru( self ):
host = '127.0.0.1'
port = HC.DEFAULT_LOCAL_BOORU_PORT
self._test_basics( host, port )
self._test_local_booru( host, port )
class TestAMP( unittest.TestCase ):
@classmethod

BIN
static/audio_resized.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.3 KiB

BIN
static/flash_resized.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.6 KiB

BIN
static/hydrus_resized.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

View File

@ -0,0 +1,25 @@
a { color: #222; text-decoration: none; font-weight: bold; }
a:hover { color: #555 }
body { font-family: "Calibri", Arial, sans-serif; color: #555; line-height: 1.5; }
h3 { color: #222; }
ul li { list-style: none; margin: 1.0em 0em; }
ul.bulletpoint li { list-style: disc; margin: 1.0em 0em; }
.dealwithit { color: #ff0000; text-shadow:
1px 1px 0 #ff7f00,
2px 2px 0 #ffff00,
3px 3px 0 #00ff00,
4px 4px 0 #0000ff,
5px 5px 0 #6600ff,
6px 6px 0 #8b00ff; }
.warning { color: #d00 }
.lololol { text-decoration: line-through; }
.right { text-align: right; }
.screenshot { float: right; clear: both; margin: 10px; }
.footer { text-align: center; font-size: 80% }
.media { clear: both; }
.thumbnail { margin: 1px; border: 1px rgb( 223, 227, 230 ) solid; display: inline-block; }
.thumbnail_container { text-align: center; display: table-cell; vertical-align: middle; }
.thumbnail_container img { display: block; margin-left: auto; margin-right: auto; }
.timeout { font-size: 80%; float: right; margin: 2px }

BIN
static/pdf_resized.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.5 KiB

BIN
static/video_resized.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.3 KiB

13
test.py
View File

@ -43,6 +43,7 @@ class App( wx.App ):
self._reads = {}
self._reads[ 'hydrus_sessions' ] = []
self._reads[ 'local_booru_share_keys' ] = []
self._reads[ 'messaging_sessions' ] = []
self._reads[ 'tag_censorship' ] = []
self._reads[ 'options' ] = CC.CLIENT_DEFAULT_OPTIONS
@ -63,13 +64,21 @@ class App( wx.App ):
self._managers[ 'tag_censorship' ] = HydrusTags.TagCensorshipManager()
self._managers[ 'tag_siblings' ] = HydrusTags.TagSiblingsManager()
self._managers[ 'tag_parents' ] = HydrusTags.TagParentsManager()
self._managers[ 'undo' ] = CC.UndoManager()
self._managers[ 'web_sessions' ] = TestConstants.FakeWebSessionManager()
self._managers[ 'restricted_services_sessions' ] = HydrusSessions.HydrusSessionManagerServer()
self._managers[ 'messaging_sessions' ] = HydrusSessions.HydrusMessagingSessionManagerServer()
info = {}
info[ 'max_monthly_data' ] = None
info[ 'used_monthly_data' ] = 0
service = CC.Service( HC.LOCAL_BOORU_SERVICE_IDENTIFIER, info )
HC.app.SetRead( 'service', service )
self._managers[ 'local_booru' ] = CC.LocalBooruCache()
self._cookies = {}
suites = []