Version 119
This commit is contained in:
Hydrus
parent
6ecbca90fd
commit
babac9e405
|
|
@ -0,0 +1,3 @@
|
|||
If you are running from source and want videos to render, stick ffmpeg.exe (for windows) or ffmpeg executable file (for linux/osx) in this folder.
|
||||
|
||||
You can get a static build at http://ffmpeg.org/download.html
|
||||
|
|
@ -17,6 +17,7 @@ from include import HydrusConstants as HC
|
|||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
from include import ClientController
|
||||
import threading
|
||||
|
|
@ -32,14 +33,20 @@ with open( HC.LOGS_DIR + os.path.sep + 'client.log', 'a' ) as f:
|
|||
|
||||
try:
|
||||
|
||||
print( 'hydrus client started at ' + time.ctime() )
|
||||
|
||||
threading.Thread( target = reactor.run, kwargs = { 'installSignalHandlers' : 0 } ).start()
|
||||
|
||||
app = ClientController.Controller()
|
||||
|
||||
app.MainLoop()
|
||||
|
||||
print( 'hydrus client shut down at ' + time.ctime() )
|
||||
|
||||
except:
|
||||
|
||||
print( 'hydrus client failed at ' + time.ctime() )
|
||||
|
||||
import traceback
|
||||
print( traceback.format_exc() )
|
||||
|
||||
|
|
|
|||
|
|
@ -8,6 +8,20 @@
|
|||
<div class="content">
|
||||
<h3>changelog</h3>
|
||||
<ul>
|
||||
<li><h3>version 119</h3></li>
|
||||
<ul>
|
||||
<li>fixed an overzealous error in v118 update code</li>
|
||||
<li>new direct ffmpeg video decoder for webm</li>
|
||||
<li>new system takes advantage of multi-core processors, increasing render speed</li>
|
||||
<li>improved animated frame timings</li>
|
||||
<li>fixed a number of gifs that were causing the "None has no 'shape' attribute" error</li>
|
||||
<li>sped up memory and subprocess cleanup after video playback</li>
|
||||
<li>added some new ffmpeg instructions to the running from source help page</li>
|
||||
<li>added client and server startup and shutdown statements to the log</li>
|
||||
<li>improved some idle time logic, which should improve maintenance timings</li>
|
||||
<li>fixed a critical idle timing bug that was adding 1s to many actions, including file queries!</li>
|
||||
<li>fixed a pubsub testing bug that was preventing daemons from quickly closing after a unit test</li>
|
||||
</ul>
|
||||
<li><h3>version 118</h3></li>
|
||||
<ul>
|
||||
<li>improved animated frame buffer size calculation, saving memory and improving smaller animation performance</li>
|
||||
|
|
|
|||
|
|
@ -16,18 +16,18 @@
|
|||
<p>This works very like hydrus repository synchronisation, in the background, reporting its progress in the middle of the status bar at the bottom of the main window, like so:</p>
|
||||
<p><img src="subs_status.png" /></p>
|
||||
<p>You don't really have to care about this all that much; it just lets you know what it is doing.</p>
|
||||
<p>Errors will be recovered from as gracefully as possible. The client will retry that subscription the next day.</p>
|
||||
<p>Serious errors, like server 404s, are recovered from as gracefully as possible. The client will retry those subscriptions the next day.</p>
|
||||
<p>Here's the result of the subscription I set up above:</p>
|
||||
<p><img src="subs_import_done.png" /></p>
|
||||
<p>It took about two minutes to download all that, and it all happened quietly in the background. Notice the 146 pending tags, up top.</p>
|
||||
<h3>how could this possibly go wrong?</h3>
|
||||
<p>This is quite a powerful tool, and if you are silly, you will end up spamming a server and likely upsetting someone or breaking something.</p>
|
||||
<p>To initialise a subscription, the daemon will parse every single gallery and image page for that particular search. This is fine for the example above, which had 4 gallery pages and 73 image pages, but the search "short hair" on safebooru has about 6,400 gallery pages encompassing >250,000 results! Trying a search like that will take a tremendous amount of time for you and cause a non-trivial CPU and data hit to their server.</p>
|
||||
<p>To initialise a subscription, the client will parse every single currently existing gallery and image page for that particular search. This is fine for the example above, which had 4 gallery pages and 73 image pages, but the search "short hair" on safebooru has about 6,400 gallery pages encompassing >250,000 results! Trying a search like that will take a tremendous amount of time for you and cause a non-trivial CPU and data hit to their server.</p>
|
||||
<p><i>Remember: If you are going to scrape anything from a site, be polite about it!</i></p>
|
||||
<p>So, I advise you start with artist searches to begin with. These usually top out at about 1,000 files, and hence don't take all that long. Once you are more confident, try doing multiple-tag queries. I suggest you leave simple single-tag queries for the manual download page, where you can hit 'that's enough' after ten or twenty pages.</p>
|
||||
<h3>help! it won't stop!</h3>
|
||||
<p>If you <i>do</i> put in a huge search, and the 'found x new files for subscription y' message is climbing terrifyingly higher and higher with no end in sight, you can pause the subscriptions daemon with <i>services->pause->subscriptions synchronisation</i>.</p>
|
||||
<p>This will give you a breather to edit your subscriptions in the dialog. Remember to unpause when you are done.</p>
|
||||
<p>This will give you a breather to reedit your subscriptions in the dialog. Remember to unpause when you are done.</p>
|
||||
<p class="right"><a href="index.html">Go back to the index ---></a></p>
|
||||
</div>
|
||||
</body>
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@
|
|||
<p>For Windows, you can do the same thing with easy_install and pip and module installers. Since it doesn't natively come with Python, you'll probably need to go through a bigger list. You'll find <a href="http://www.lfd.uci.edu/~gohlke/pythonlibs/">this</a> page very helpful. I have a fair bit of experience with Windows python, so send me a mail if you need help.</a>
|
||||
<p>Some people have encountered problems with wxPython 3.0, so I use 2.9.x in my official releases. Again, YMMV.</p>
|
||||
<p>You'll probably want to install Pillow (pip should do it for Linux/OS X, easy_install for Windows) instead of PIL, but go back to PIL if Pillow doesn't work.</p>
|
||||
<p>If you want to import videos, you will need to put a static <a href="http://ffmpeg.org/">FFMPEG</a> executable in the install_dir/bin directory. Have a look at how I do it in the extractable compiled releases if you can't figure it out. You can either copy the exe from one of those releases, or download it right from the FFMPEG site. I don't include the FFMPEG exes in the source release like I do upnpc just because they are so big.</a>
|
||||
<p>Once you have everything set up, client.pyw and server.pyw should look for and run off the database files just like the executables.</p>
|
||||
<p>I develop hydrus on 64-bit Win 7, so the program is much more stable and reasonable on Windows. I am developing for Linux and OS X, but I do not have as much experience with them, so I would particularly appreciate your Linux/OS X bug reports and any informed suggestions.</p>
|
||||
<h3>my code</h3>
|
||||
|
|
|
|||
|
|
@ -122,10 +122,10 @@ The database will be locked while the backup occurs, which may lock up your gui
|
|||
|
||||
def EventPubSub( self, event ):
|
||||
|
||||
HC.busy_doing_pubsub = True
|
||||
HC.currently_doing_pubsub = True
|
||||
|
||||
try: HC.pubsub.WXProcessQueueItem()
|
||||
finally: HC.busy_doing_pubsub = False
|
||||
finally: HC.currently_doing_pubsub = False
|
||||
|
||||
|
||||
def GetFullscreenImageCache( self ): return self._fullscreen_image_cache
|
||||
|
|
@ -331,7 +331,7 @@ The database will be locked while the backup occurs, which may lock up your gui
|
|||
|
||||
def Read( self, action, *args, **kwargs ):
|
||||
|
||||
self._last_idle_time = HC.GetNow()
|
||||
if action == 'media_results': self._last_idle_time = HC.GetNow()
|
||||
|
||||
return self._Read( action, *args, **kwargs )
|
||||
|
||||
|
|
@ -506,7 +506,7 @@ Once it is done, the client will restart.'''
|
|||
|
||||
def TIMEREventMaintenance( self, event ):
|
||||
|
||||
del gc.garbage[:]
|
||||
gc.collect()
|
||||
|
||||
if self.CurrentlyIdle(): self.MaintainDB()
|
||||
|
||||
|
|
@ -518,12 +518,7 @@ Once it is done, the client will restart.'''
|
|||
while True:
|
||||
|
||||
if HC.shutdown: raise Exception( 'Client shutting down!' )
|
||||
elif HC.pubsub.NotBusy() and not HC.busy_doing_pubsub:
|
||||
|
||||
if not self.CurrentlyIdle(): time.sleep( 1 )
|
||||
|
||||
return
|
||||
|
||||
elif HC.pubsub.NoJobsQueued() and not HC.currently_doing_pubsub: return
|
||||
else: time.sleep( 0.0001 )
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -4897,13 +4897,17 @@ class DB( ServiceDB ):
|
|||
|
||||
hash = filename.decode( 'hex' )
|
||||
|
||||
phash = HydrusImageHandling.GeneratePerceptualHash( path )
|
||||
|
||||
hash_id = self._GetHashId( c, hash )
|
||||
|
||||
c.execute( 'INSERT OR REPLACE INTO perceptual_hashes ( hash_id, phash ) VALUES ( ?, ? );', ( hash_id, sqlite3.Binary( phash ) ) )
|
||||
|
||||
i += 1
|
||||
try:
|
||||
|
||||
phash = HydrusImageHandling.GeneratePerceptualHash( path )
|
||||
|
||||
hash_id = self._GetHashId( c, hash )
|
||||
|
||||
c.execute( 'INSERT OR REPLACE INTO perceptual_hashes ( hash_id, phash ) VALUES ( ?, ? );', ( hash_id, sqlite3.Binary( phash ) ) )
|
||||
|
||||
i += 1
|
||||
|
||||
except: print( 'When updating to v118, ' + path + '\'s phash could not be recalculated.' )
|
||||
|
||||
if i % 100 == 0: HC.app.SetSplashText( 'reprocessing thumbs: ' + HC.ConvertIntToPrettyString( i ) )
|
||||
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import ClientGUIDialogs
|
|||
import ClientGUIDialogsManage
|
||||
import ClientGUIMixins
|
||||
import collections
|
||||
import gc
|
||||
import HydrusImageHandling
|
||||
import HydrusVideoHandling
|
||||
import os
|
||||
|
|
@ -78,8 +79,6 @@ def GetExtraDimensions( media ):
|
|||
|
||||
class Animation( wx.Window ):
|
||||
|
||||
UPDATE_MS = 16
|
||||
|
||||
def __init__( self, parent, media, initial_size, initial_position ):
|
||||
|
||||
wx.Window.__init__( self, parent, size = initial_size, pos = initial_position )
|
||||
|
|
@ -110,7 +109,12 @@ class Animation( wx.Window ):
|
|||
|
||||
self.EventResize( None )
|
||||
|
||||
self._timer_video.Start( self.UPDATE_MS, wx.TIMER_CONTINUOUS )
|
||||
self._timer_video.Start( 16, wx.TIMER_ONE_SHOT )
|
||||
|
||||
|
||||
def __del__( self ):
|
||||
|
||||
wx.CallLater( 500, gc.collect )
|
||||
|
||||
|
||||
def _DrawFrame( self ):
|
||||
|
|
@ -140,7 +144,11 @@ class Animation( wx.Window ):
|
|||
|
||||
self._current_frame_drawn = True
|
||||
|
||||
self._current_frame_drawn_at = max( time.clock(), self._current_frame_drawn_at + ( self._video_container.GetDuration( self._current_frame_index ) / 1000 ) )
|
||||
now_in_ms = time.clock()
|
||||
frame_was_supposed_to_be_at = self._current_frame_drawn_at + ( self._video_container.GetDuration( self._current_frame_index ) / 1000 )
|
||||
|
||||
if 1000.0 * ( now_in_ms - frame_was_supposed_to_be_at ) > 16.7: self._current_frame_drawn_at = now_in_ms
|
||||
else: self._current_frame_drawn_at = frame_was_supposed_to_be_at
|
||||
|
||||
|
||||
def _DrawWhite( self ):
|
||||
|
|
@ -227,13 +235,15 @@ class Animation( wx.Window ):
|
|||
|
||||
def TIMEREventVideo( self, event ):
|
||||
|
||||
MIN_TIMER_TIME = 4
|
||||
|
||||
if self.IsShown():
|
||||
|
||||
if self._current_frame_drawn:
|
||||
|
||||
ms_since_current_frame_drawn = int( 1000.0 * ( time.clock() - self._current_frame_drawn_at ) )
|
||||
|
||||
time_to_update = ms_since_current_frame_drawn + float( self.UPDATE_MS ) / 2 > self._video_container.GetDuration( self._current_frame_index )
|
||||
time_to_update = ms_since_current_frame_drawn + MIN_TIMER_TIME / 2 > self._video_container.GetDuration( self._current_frame_index )
|
||||
|
||||
if not self._paused and time_to_update:
|
||||
|
||||
|
|
@ -249,6 +259,12 @@ class Animation( wx.Window ):
|
|||
|
||||
if not self._current_frame_drawn and self._video_container.HasFrame( self._current_frame_index ): self._DrawFrame()
|
||||
|
||||
ms_since_current_frame_drawn = int( 1000.0 * ( time.clock() - self._current_frame_drawn_at ) )
|
||||
|
||||
ms_until_next_frame = max( MIN_TIMER_TIME, self._video_container.GetDuration( self._current_frame_index ) - ms_since_current_frame_drawn )
|
||||
|
||||
self._timer_video.Start( ms_until_next_frame, wx.TIMER_ONE_SHOT )
|
||||
|
||||
|
||||
|
||||
class AnimationBar( wx.Window ):
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ options = {}
|
|||
# Misc
|
||||
|
||||
NETWORK_VERSION = 13
|
||||
SOFTWARE_VERSION = 118
|
||||
SOFTWARE_VERSION = 119
|
||||
|
||||
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )
|
||||
|
||||
|
|
@ -84,7 +84,7 @@ subs_changed = False
|
|||
|
||||
http = None
|
||||
|
||||
busy_doing_pubsub = False
|
||||
currently_doing_pubsub = False
|
||||
|
||||
# Enums
|
||||
|
||||
|
|
|
|||
|
|
@ -1135,7 +1135,7 @@ class ImportArgsGeneratorGallery( ImportArgsGenerator ):
|
|||
|
||||
if status == 'redundant':
|
||||
|
||||
( media_result, ) = HC.app.Read( 'media_results', HC.LOCAL_FILE_SERVICE_IDENTIFIER, ( hash, ) )
|
||||
( media_result, ) = HC.app.ReadDaemon( 'media_results', HC.LOCAL_FILE_SERVICE_IDENTIFIER, ( hash, ) )
|
||||
|
||||
do_tags = len( self._advanced_tag_options ) > 0
|
||||
|
||||
|
|
@ -1263,7 +1263,7 @@ class ImportArgsGeneratorThread( ImportArgsGenerator ):
|
|||
|
||||
if status == 'redundant':
|
||||
|
||||
( media_result, ) = HC.app.Read( 'media_results', HC.LOCAL_FILE_SERVICE_IDENTIFIER, ( hash, ) )
|
||||
( media_result, ) = HC.app.ReadDaemon( 'media_results', HC.LOCAL_FILE_SERVICE_IDENTIFIER, ( hash, ) )
|
||||
|
||||
return ( status, media_result )
|
||||
|
||||
|
|
@ -1303,7 +1303,7 @@ class ImportArgsGeneratorURLs( ImportArgsGenerator ):
|
|||
|
||||
if status == 'redundant':
|
||||
|
||||
( media_result, ) = HC.app.Read( 'media_results', HC.LOCAL_FILE_SERVICE_IDENTIFIER, ( hash, ) )
|
||||
( media_result, ) = HC.app.ReadDaemon( 'media_results', HC.LOCAL_FILE_SERVICE_IDENTIFIER, ( hash, ) )
|
||||
|
||||
return ( status, media_result )
|
||||
|
||||
|
|
|
|||
|
|
@ -523,7 +523,7 @@ class HydrusBitmap():
|
|||
|
||||
def GetSize( self ): return self._size
|
||||
|
||||
class RasterContainer():
|
||||
class RasterContainer( object ):
|
||||
|
||||
def __init__( self, media, target_resolution = None ):
|
||||
|
||||
|
|
|
|||
|
|
@ -60,7 +60,7 @@ class HydrusPubSub():
|
|||
return callables
|
||||
|
||||
|
||||
def NotBusy( self ):
|
||||
def NoJobsQueued( self ):
|
||||
|
||||
with self._lock:
|
||||
|
||||
|
|
@ -131,7 +131,7 @@ class HydrusPubSub():
|
|||
def WXpubimmediate( self, topic, *args, **kwargs ):
|
||||
|
||||
with self._lock:
|
||||
|
||||
|
||||
callables = self._GetCallables( topic )
|
||||
|
||||
for callable in callables: callable( *args, **kwargs )
|
||||
|
|
|
|||
|
|
@ -140,6 +140,8 @@ class DAEMONCallToThread( DAEMON ):
|
|||
|
||||
callable( *args, **kwargs )
|
||||
|
||||
del callable
|
||||
|
||||
except Exception as e: HC.ShowException( e )
|
||||
|
||||
time.sleep( 0.00001 )
|
||||
|
|
|
|||
|
|
@ -7,12 +7,19 @@ import HydrusExceptions
|
|||
import HydrusImageHandling
|
||||
import HydrusThreading
|
||||
import matroska
|
||||
import numpy
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import traceback
|
||||
import threading
|
||||
import time
|
||||
from wx import wx
|
||||
|
||||
if HC.PLATFORM_LINUX: FFMPEG_PATH = '"' + HC.BIN_DIR + os.path.sep + 'ffmpeg"'
|
||||
elif HC.PLATFORM_OSX: FFMPEG_PATH = '"' + HC.BIN_DIR + os.path.sep + 'ffmpeg"'
|
||||
elif HC.PLATFORM_WINDOWS: FFMPEG_PATH = '"' + HC.BIN_DIR + os.path.sep + 'ffmpeg.exe"'
|
||||
|
||||
def GetCVVideoProperties( path ):
|
||||
|
||||
capture = cv2.VideoCapture( path )
|
||||
|
|
@ -120,6 +127,247 @@ def GetMatroskaOrWebMProperties( path ):
|
|||
|
||||
return ( ( width, height ), duration, num_frames )
|
||||
|
||||
# This is cribbed from moviepy's FFMPEG_VideoReader
|
||||
class HydrusFFMPEG_VideoReader( object ):
|
||||
|
||||
def __init__(self, media, print_infos=False, bufsize = None, pix_fmt="rgb24"):
|
||||
|
||||
self._media = media
|
||||
|
||||
hash = self._media.GetHash()
|
||||
mime = self._media.GetMime()
|
||||
|
||||
self._path = CC.GetFilePath( hash, mime )
|
||||
|
||||
self.size = self._media.GetResolution()
|
||||
self.duration = float( self._media.GetDuration() ) / 1000.0
|
||||
self.nframes = self._media.GetNumFrames()
|
||||
|
||||
self.fps = float( self.nframes ) / self.duration
|
||||
|
||||
self.pix_fmt = pix_fmt
|
||||
|
||||
if pix_fmt == 'rgba': self.depth = 4
|
||||
else: self.depth = 3
|
||||
|
||||
if bufsize is None:
|
||||
|
||||
( x, y ) = self.size
|
||||
bufsize = self.depth * x * y * 5
|
||||
|
||||
|
||||
self.process = None
|
||||
|
||||
self.bufsize = bufsize
|
||||
|
||||
self.initialize()
|
||||
|
||||
|
||||
def __del__( self ):
|
||||
|
||||
self.close()
|
||||
|
||||
|
||||
def close(self):
|
||||
|
||||
if self.process is not None:
|
||||
|
||||
self.process.terminate()
|
||||
|
||||
self.process.stdout.close()
|
||||
self.process.stderr.close()
|
||||
|
||||
self.process = None
|
||||
|
||||
|
||||
|
||||
def initialize( self, starttime=0 ):
|
||||
"""Opens the file, creates the pipe. """
|
||||
|
||||
self.close()
|
||||
|
||||
if starttime != 0:
|
||||
|
||||
offset = min( 1, starttime )
|
||||
|
||||
i_arg = [ '-ss', "%.03f" % ( starttime - offset ), '-i', '"' + self._path + '"' ]
|
||||
|
||||
else: i_arg = [ '-i', '"' + self._path + '"' ]
|
||||
|
||||
cmd = ([FFMPEG_PATH]+ i_arg +
|
||||
['-loglevel', 'error',
|
||||
'-f', 'image2pipe',
|
||||
"-pix_fmt", self.pix_fmt,
|
||||
'-vcodec', 'rawvideo', '-'])
|
||||
|
||||
self.process = subprocess.Popen( ' '.join( cmd ), shell = True, bufsize= self.bufsize, stdout=subprocess.PIPE, stderr=subprocess.PIPE )
|
||||
|
||||
self.pos = int( round( self.fps * starttime ) )
|
||||
|
||||
|
||||
def skip_frames(self, n=1):
|
||||
|
||||
"""Reads and throws away n frames """
|
||||
|
||||
( w, h ) = self.size
|
||||
|
||||
for i in range( n ):
|
||||
|
||||
self.process.stdout.read( self.depth * w * h )
|
||||
self.process.stdout.flush()
|
||||
|
||||
|
||||
self.pos += n
|
||||
|
||||
|
||||
def read_frame( self ):
|
||||
|
||||
( w, h ) = self.size
|
||||
|
||||
nbytes = self.depth * w * h
|
||||
|
||||
s = self.process.stdout.read(nbytes)
|
||||
|
||||
self.pos += 1
|
||||
|
||||
if len(s) != nbytes:
|
||||
|
||||
print( "Warning: in file %s, "%(self._path)+
|
||||
"%d bytes wanted but %d bytes read,"%(nbytes, len(s))+
|
||||
"at frame %d/%d, at time %.02f/%.02f sec. "%(
|
||||
self.pos,self.nframes,
|
||||
1.0*self.pos/self.fps,
|
||||
self.duration)+
|
||||
"Using the last valid frame instead.")
|
||||
result = self.lastread
|
||||
|
||||
self.close()
|
||||
|
||||
else:
|
||||
|
||||
result = numpy.fromstring( s, dtype = 'uint8' ).reshape( ( h, w, len( s ) // ( w * h ) ) )
|
||||
|
||||
self.lastread = result
|
||||
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def set_position( self, pos ):
|
||||
|
||||
if ( pos < self.pos ) or ( pos > self.pos + 60 ):
|
||||
|
||||
starttime = float( pos ) / self.fps
|
||||
|
||||
self.initialize( starttime )
|
||||
|
||||
else: self.skip_frames( pos - self.pos )
|
||||
|
||||
|
||||
# same thing; this is cribbed from moviepy
|
||||
def Hydrusffmpeg_parse_infos(filename, print_infos=False):
|
||||
"""Get file infos using ffmpeg.
|
||||
|
||||
Returns a dictionnary with the fields:
|
||||
"video_found", "video_fps", "duration", "video_nframes",
|
||||
"video_duration"
|
||||
"audio_found", "audio_fps"
|
||||
|
||||
"video_duration" is slightly smaller than "duration" to avoid
|
||||
fetching the uncomplete frames at the end, which raises an error.
|
||||
|
||||
"""
|
||||
|
||||
# open the file in a pipe, provoke an error, read output
|
||||
|
||||
cmd = [FFMPEG_PATH, "-i", '"' + filename + '"']
|
||||
|
||||
is_GIF = filename.endswith('.gif')
|
||||
|
||||
if is_GIF:
|
||||
if HC.PLATFORM_WINDOWS: cmd += ["-f", "null", "NUL"]
|
||||
else: cmd += ["-f", "null", "/dev/null"]
|
||||
|
||||
proc = subprocess.Popen( ' '.join( cmd ), shell = True, bufsize=10**5, stdout=subprocess.PIPE, stderr=subprocess.PIPE )
|
||||
|
||||
infos = proc.stderr.read().decode('utf8')
|
||||
proc.terminate()
|
||||
|
||||
del proc
|
||||
|
||||
if print_infos:
|
||||
# print the whole info text returned by FFMPEG
|
||||
print( infos )
|
||||
|
||||
lines = infos.splitlines()
|
||||
if "No such file or directory" in lines[-1]:
|
||||
raise IOError("%s not found ! Wrong path ?"%filename)
|
||||
|
||||
result = dict()
|
||||
|
||||
# get duration (in seconds)
|
||||
try:
|
||||
keyword = ('frame=' if is_GIF else 'Duration: ')
|
||||
line = [l for l in lines if keyword in l][0]
|
||||
match = re.search("[0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9][0-9]", line)
|
||||
hms = map(float, line[match.start()+1:match.end()].split(':'))
|
||||
|
||||
if len(hms) == 1:
|
||||
result['duration'] = hms[0]
|
||||
elif len(hms) == 2:
|
||||
result['duration'] = 60*hms[0]+hms[1]
|
||||
elif len(hms) ==3:
|
||||
result['duration'] = 3600*hms[0]+60*hms[1]+hms[2]
|
||||
|
||||
except:
|
||||
raise IOError("Error reading duration in file %s,"%(filename)+
|
||||
"Text parsed: %s"%infos)
|
||||
|
||||
# get the output line that speaks about video
|
||||
lines_video = [l for l in lines if ' Video: ' in l]
|
||||
|
||||
result['video_found'] = ( lines_video != [] )
|
||||
|
||||
if result['video_found']:
|
||||
|
||||
line = lines_video[0]
|
||||
|
||||
# get the size, of the form 460x320 (w x h)
|
||||
match = re.search(" [0-9]*x[0-9]*(,| )", line)
|
||||
s = list(map(int, line[match.start():match.end()-1].split('x')))
|
||||
result['video_size'] = s
|
||||
|
||||
|
||||
# get the frame rate
|
||||
try:
|
||||
match = re.search("( [0-9]*.| )[0-9]* tbr", line)
|
||||
result['video_fps'] = float(line[match.start():match.end()].split(' ')[1])
|
||||
except:
|
||||
match = re.search("( [0-9]*.| )[0-9]* fps", line)
|
||||
result['video_fps'] = float(line[match.start():match.end()].split(' ')[1])
|
||||
|
||||
result['video_nframes'] = int(result['duration']*result['video_fps'])+1
|
||||
|
||||
result['video_duration'] = result['duration']
|
||||
# We could have also recomputed the duration from the number
|
||||
# of frames, as follows:
|
||||
# >>> result['video_duration'] = result['video_nframes'] / result['video_fps']
|
||||
|
||||
|
||||
lines_audio = [l for l in lines if ' Audio: ' in l]
|
||||
|
||||
result['audio_found'] = lines_audio != []
|
||||
|
||||
if result['audio_found']:
|
||||
line = lines_audio[0]
|
||||
try:
|
||||
match = re.search(" [0-9]* Hz", line)
|
||||
result['audio_fps'] = int(line[match.start()+1:match.end()])
|
||||
except:
|
||||
result['audio_fps'] = 'unknown'
|
||||
|
||||
return result
|
||||
|
||||
class VideoContainer( HydrusImageHandling.RasterContainer ):
|
||||
|
||||
BUFFER_SIZE = 1024 * 1024 * 96
|
||||
|
|
@ -143,7 +391,7 @@ class VideoContainer( HydrusImageHandling.RasterContainer ):
|
|||
if self._media.GetMime() == HC.IMAGE_GIF: self._durations = HydrusImageHandling.GetGIFFrameDurations( self._path )
|
||||
else: self._frame_duration = GetVideoFrameDuration( self._path )
|
||||
|
||||
self._renderer = VideoRenderer( self, self._media, self._target_resolution )
|
||||
self._renderer = VideoRendererMoviePy( self, self._media, self._target_resolution )
|
||||
|
||||
num_frames = self.GetNumFrames()
|
||||
|
||||
|
|
@ -231,16 +479,16 @@ class VideoContainer( HydrusImageHandling.RasterContainer ):
|
|||
|
||||
self._maximum_frame_asked_for = new_maximum_frame_to_ask_for
|
||||
|
||||
self._renderer.SetRenderToPosition( self._maximum_frame_asked_for)
|
||||
self._renderer.SetRenderToPosition( self._maximum_frame_asked_for )
|
||||
|
||||
|
||||
else:
|
||||
else: # index > self._last_index_asked_for
|
||||
|
||||
no_wraparound = self._minimum_frame_asked_for < self._maximum_frame_asked_for
|
||||
currently_no_wraparound = self._minimum_frame_asked_for < self._maximum_frame_asked_for
|
||||
|
||||
self._minimum_frame_asked_for = new_minimum_frame_to_ask_for
|
||||
|
||||
if no_wraparound:
|
||||
if currently_no_wraparound:
|
||||
|
||||
if index > self._maximum_frame_asked_for:
|
||||
|
||||
|
|
@ -268,8 +516,9 @@ class VideoContainer( HydrusImageHandling.RasterContainer ):
|
|||
|
||||
|
||||
self._MaintainBuffer()
|
||||
|
||||
|
||||
class VideoRenderer():
|
||||
class VideoRendererCV():
|
||||
|
||||
def __init__( self, image_container, media, target_resolution ):
|
||||
|
||||
|
|
@ -419,4 +668,142 @@ class VideoRenderer():
|
|||
|
||||
|
||||
|
||||
|
||||
class VideoRendererMoviePy():
|
||||
|
||||
def __init__( self, image_container, media, target_resolution ):
|
||||
|
||||
self._image_container = image_container
|
||||
self._media = media
|
||||
self._target_resolution = target_resolution
|
||||
|
||||
self._lock = threading.Lock()
|
||||
|
||||
( x, y ) = media.GetResolution()
|
||||
|
||||
self._ffmpeg_reader = HydrusFFMPEG_VideoReader( media, bufsize = x * y * 3 * 5 )
|
||||
|
||||
self._current_index = 0
|
||||
self._last_index_rendered = -1
|
||||
self._last_frame = None
|
||||
self._render_to_index = -1
|
||||
|
||||
|
||||
def _GetCurrentFrame( self ):
|
||||
|
||||
try: image = self._ffmpeg_reader.read_frame()
|
||||
except Exception as e: raise HydrusExceptions.CantRenderWithCVException( 'FFMPEG could not render frame ' + HC.u( self._current_index ) + '.' + os.linesep * 2 + HC.u( e ) )
|
||||
|
||||
self._last_index_rendered = self._current_index
|
||||
|
||||
num_frames = self._media.GetNumFrames()
|
||||
|
||||
self._current_index = ( self._current_index + 1 ) % num_frames
|
||||
|
||||
if self._current_index == 0 and self._last_index_rendered != 0: self._ffmpeg_reader.initialize()
|
||||
|
||||
return image
|
||||
|
||||
|
||||
def _RenderCurrentFrame( self ):
|
||||
|
||||
try:
|
||||
|
||||
image = self._GetCurrentFrame()
|
||||
|
||||
image = HydrusImageHandling.EfficientlyResizeCVImage( image, self._target_resolution )
|
||||
|
||||
self._last_frame = image
|
||||
|
||||
except HydrusExceptions.CantRenderWithCVException:
|
||||
|
||||
if self._last_frame is None: raise
|
||||
|
||||
image = self._last_frame
|
||||
|
||||
|
||||
return HydrusImageHandling.GenerateHydrusBitmapFromCVImage( image )
|
||||
|
||||
|
||||
def _RenderFrames( self ):
|
||||
|
||||
no_frames_yet = True
|
||||
|
||||
while True:
|
||||
|
||||
try:
|
||||
|
||||
yield self._RenderCurrentFrame()
|
||||
|
||||
no_frames_yet = False
|
||||
|
||||
except HydrusExceptions.CantRenderWithCVException:
|
||||
|
||||
if no_frames_yet: raise
|
||||
else: break
|
||||
|
||||
|
||||
|
||||
|
||||
def SetRenderToPosition( self, index ):
|
||||
|
||||
with self._lock:
|
||||
|
||||
if self._render_to_index != index:
|
||||
|
||||
self._render_to_index = index
|
||||
|
||||
HydrusThreading.CallToThread( self.THREADDoWork )
|
||||
|
||||
|
||||
|
||||
|
||||
def SetPosition( self, index ):
|
||||
|
||||
with self._lock:
|
||||
|
||||
if index == self._current_index: return
|
||||
else:
|
||||
|
||||
if self._media.GetMime() == HC.IMAGE_GIF:
|
||||
|
||||
if index < self._current_index:
|
||||
|
||||
self._ffmpeg_reader.initialize()
|
||||
|
||||
self._ffmpeg_reader.skip_frames( index )
|
||||
|
||||
|
||||
else:
|
||||
|
||||
timecode = float( index ) / self._ffmpeg_reader.fps
|
||||
|
||||
self._ffmpeg_reader.set_position( timecode )
|
||||
|
||||
|
||||
self._render_to_index = index
|
||||
|
||||
|
||||
|
||||
|
||||
def THREADDoWork( self ):
|
||||
|
||||
while True:
|
||||
|
||||
time.sleep( 0.00001 ) # thread yield
|
||||
|
||||
with self._lock:
|
||||
|
||||
if self._last_index_rendered != self._render_to_index:
|
||||
|
||||
index = self._current_index
|
||||
|
||||
frame = self._RenderCurrentFrame()
|
||||
|
||||
wx.CallAfter( self._image_container.AddFrame, index, frame )
|
||||
|
||||
else: break
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -5,6 +5,7 @@ import HydrusTags
|
|||
import os
|
||||
import random
|
||||
import threading
|
||||
import weakref
|
||||
|
||||
tinest_gif = '\x47\x49\x46\x38\x39\x61\x01\x00\x01\x00\x00\xFF\x00\x2C\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x00\x3B'
|
||||
|
||||
|
|
@ -60,14 +61,58 @@ class FakePubSub():
|
|||
|
||||
self._pubsubs = collections.defaultdict( list )
|
||||
|
||||
self._callables = []
|
||||
|
||||
self._lock = threading.Lock()
|
||||
|
||||
self._topics_to_objects = {}
|
||||
self._topics_to_method_names = {}
|
||||
|
||||
|
||||
def _GetCallables( self, topic ):
|
||||
|
||||
callables = []
|
||||
|
||||
if topic in self._topics_to_objects:
|
||||
|
||||
try:
|
||||
|
||||
objects = self._topics_to_objects[ topic ]
|
||||
|
||||
for object in objects:
|
||||
|
||||
method_names = self._topics_to_method_names[ topic ]
|
||||
|
||||
for method_name in method_names:
|
||||
|
||||
if hasattr( object, method_name ):
|
||||
|
||||
try:
|
||||
|
||||
callable = getattr( object, method_name )
|
||||
|
||||
callables.append( callable )
|
||||
|
||||
except wx.PyDeadObjectError: pass
|
||||
except TypeError as e:
|
||||
|
||||
if '_wxPyDeadObject' not in str( e ): raise
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
except: pass
|
||||
|
||||
|
||||
return callables
|
||||
|
||||
|
||||
def ClearPubSubs( self ): self._pubsubs = collections.defaultdict( list )
|
||||
|
||||
def GetPubSubs( self, topic ): return self._pubsubs[ topic ]
|
||||
|
||||
def NotBusy( self ): return True
|
||||
def NoJobsQueued( self ): return True
|
||||
|
||||
def WXProcessQueueItem( self ): pass
|
||||
|
||||
|
|
@ -79,13 +124,22 @@ class FakePubSub():
|
|||
|
||||
|
||||
|
||||
def sub( self, object, method_name, topic ): pass
|
||||
def sub( self, object, method_name, topic ):
|
||||
|
||||
if topic not in self._topics_to_objects: self._topics_to_objects[ topic ] = weakref.WeakSet()
|
||||
if topic not in self._topics_to_method_names: self._topics_to_method_names[ topic ] = set()
|
||||
|
||||
self._topics_to_objects[ topic ].add( object )
|
||||
self._topics_to_method_names[ topic ].add( method_name )
|
||||
|
||||
|
||||
def WXpubimmediate( self, topic, *args, **kwargs ):
|
||||
|
||||
with self._lock:
|
||||
|
||||
self._pubsubs[ topic ].append( ( args, kwargs ) )
|
||||
callables = self._GetCallables( topic )
|
||||
|
||||
for callable in callables: callable( *args, **kwargs )
|
||||
|
||||
|
||||
|
||||
|
|
@ -17,6 +17,7 @@ from include import HydrusConstants as HC
|
|||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from include import ServerController
|
||||
import threading
|
||||
from twisted.internet import reactor
|
||||
|
|
@ -31,14 +32,20 @@ with open( HC.LOGS_DIR + os.path.sep + 'server.log', 'a' ) as f:
|
|||
|
||||
try:
|
||||
|
||||
print( 'hydrus server started at ' + time.ctime() )
|
||||
|
||||
threading.Thread( target = reactor.run, kwargs = { 'installSignalHandlers' : 0 } ).start()
|
||||
|
||||
app = ServerController.Controller()
|
||||
|
||||
app.MainLoop()
|
||||
|
||||
print( 'hydrus server shut down at ' + time.ctime() )
|
||||
|
||||
except:
|
||||
|
||||
print( 'hydrus server failed at ' + time.ctime() )
|
||||
|
||||
import traceback
|
||||
print( traceback.format_exc() )
|
||||
|
||||
|
|
|
|||
8
test.py
8
test.py
|
|
@ -63,6 +63,7 @@ class App( wx.App ):
|
|||
self._managers[ 'tag_censorship' ] = HydrusTags.TagCensorshipManager()
|
||||
self._managers[ 'tag_siblings' ] = HydrusTags.TagSiblingsManager()
|
||||
self._managers[ 'tag_parents' ] = HydrusTags.TagParentsManager()
|
||||
|
||||
self._managers[ 'undo' ] = CC.UndoManager()
|
||||
self._managers[ 'web_sessions' ] = TestConstants.FakeWebSessionManager()
|
||||
|
||||
|
|
@ -150,11 +151,14 @@ if __name__ == '__main__':
|
|||
|
||||
else: only_run = None
|
||||
|
||||
old_pubsub = HC.pubsub
|
||||
|
||||
app = App()
|
||||
|
||||
HC.shutdown = True
|
||||
|
||||
HC.pubsub.WXpubimmediate( 'shutdown' )
|
||||
|
||||
raw_input()
|
||||
|
||||
old_pubsub.WXpubimmediate( 'shutdown' )
|
||||
|
||||
raw_input()
|
||||
Loading…
Reference in New Issue