Version 294
This commit is contained in:
Hydrus Network Developer
parent
9398abfc50
commit
eccc185fcf
|
|
@ -8,6 +8,30 @@
|
|||
<div class="content">
|
||||
<h3>changelog</h3>
|
||||
<ul>
|
||||
<li><h3>version 294</h3></li>
|
||||
<ul>
|
||||
<li>fixed video scan</li>
|
||||
<li>fixed up some import folder logic--they now run 'look for new files' checks separate from 'import anything still in the queue', so they can now catch up on outstanding files more easily</li>
|
||||
<li>the ten minute file-processing break is reverted--import folders now just 'save' every ten minutes, to forestall lost work on a crash</li>
|
||||
<li>import folders now have an explicit 'check regularly' checkbox to control whether you want it to check regularly or only when you tell it to. the paused status now means 'do not ever work'</li>
|
||||
<li>import folders now make a 'working' popup, like subscriptions do, that shows new file discovery progress and found file import progress. it has a cancel button that will stop the current job and can be hidden in the import folder's options. existing import folders will default to ON for this, so you'll likely see one of these right after you update</li>
|
||||
<li>import folders can now publish their files to a new page! this effectively cuts out the button step. furthermore, if a page already exists with the import folder's name (e.g. a previous page the import folder created is still around), the import folder will publish the new files to _that_ page, updating it</li>
|
||||
<li>file popup buttons with the same name will now 'merge'! multiple work cycles of a subscription or import folder will now just update the first button rather than spamming several</li>
|
||||
<li>running import folders will respond faster to a client shutdown event</li>
|
||||
<li>wrote some better controller-level thread management, including surplus thread deletion--idle CPU should be reduced on import-busy clients</li>
|
||||
<li>wrote some code to deal with sub-second times a bit better in certain places</li>
|
||||
<li>finished off a new scheduled job queue that collapses an old multiple-thread-when-idle system down into just one</li>
|
||||
<li>many things are moved to the new job scheduling system--all the old calllater calls, and the popup message timers as well</li>
|
||||
<li>completely removed the old wx timers the autocomplete input was using, as they were just too much of a hassle. any crashing these were causing is now gone--it all works on the simpler scheduled job queue now</li>
|
||||
<li>the autocomplete dropdowns are better at judging when to engage their timers and so now use far less idle CPU</li>
|
||||
<li>fixed a crash that could occur sometime after starting a duplicate filter maintenance task from the dupe filter page</li>
|
||||
<li>fixed several unlikely but plausible crashes on the admin-side of petition processing</li>
|
||||
<li>wrote a bunch of help for the url classes and new parsing system--it isn't finished yet, but then neither is the new system!</li>
|
||||
<li>if the client fails to initialise the db, it will now try to present the error in a bit of screenshottable-gui before the program quits</li>
|
||||
<li>improved thread watcher error handling when the given url is unwatchable</li>
|
||||
<li>lots of timer related cleanup and tiny fixes</li>
|
||||
<li>mix fixes</li>
|
||||
</ul>
|
||||
<li><h3>version 293</h3></li>
|
||||
<ul>
|
||||
<li>fixed the issue with options dialog not opening again after a save--I apologise for the inconvenience</li>
|
||||
|
|
|
|||
|
|
@ -0,0 +1,14 @@
|
|||
<html>
|
||||
<head>
|
||||
<title>downloader - putting it all together</title>
|
||||
<link href="hydrus.ico" rel="shortcut icon" />
|
||||
<link href="style.css" rel="stylesheet" type="text/css" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="content">
|
||||
<p><a href="downloader_downloaders.html"><---- Back to Downloaders</a></p>
|
||||
<h3>putting it all together</h3>
|
||||
<p class="right"><a href="downloader_sharing.html">Now let's share our downloaders ----></a></p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
<html>
|
||||
<head>
|
||||
<title>downloader - downloaders</title>
|
||||
<link href="hydrus.ico" rel="shortcut icon" />
|
||||
<link href="style.css" rel="stylesheet" type="text/css" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="content">
|
||||
<p><a href="downloader_parsers.html"><---- Back to Parsers</a></p>
|
||||
<h3>downloaders</h3>
|
||||
<p class="right"><a href="downloader_completion.html">Now let's put it all together ----></a></p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 73 KiB |
|
|
@ -0,0 +1,43 @@
|
|||
<html>
|
||||
<head>
|
||||
<title>downloader - intro</title>
|
||||
<link href="hydrus.ico" rel="shortcut icon" />
|
||||
<link href="style.css" rel="stylesheet" type="text/css" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="content">
|
||||
<p class="warning">This system and help is all under construction! Even when it is done, it will be for advanced users who understand HTML or JSON. Beware!</p>
|
||||
<h3>this system</h3>
|
||||
<p>The first versions of hydrus's downloaders were all hardcoded and static--I wrote everything into the program itself and nothing was user-creatable or -fixable. After the maintenance burden of the entire messy system proved too large for me to keep up with and a semi-editable booru system proved successful, I decided to significantly overhaul the whole thing. The new system allows user creation and sharing of every component. It is designed to be very simple to the front-end user--they will typically handle a couple of png files and then select a new downloader from a list--but very flexible (and hence pretty complicated) on the back-end. These help pages describe the different compontents with the intention of making an HTML- or JSON- fluent user able to create and share a full new downloader on their own.</p>
|
||||
<p>As always, this is all under active development. Your feedback on the system would be appreciated, and if something is confusing or you discover something in here that is out of date, please <a href="contact.html">let me know</a>.</p>
|
||||
<h3>what is a downloader?</h3>
|
||||
<p>In hydrus, a downloader is one of:</p>
|
||||
<ul>
|
||||
<li><h3>Gallery Downloader</h3></li>
|
||||
<li>This takes a string like 'blue_eyes' to produce a series of thumbnail gallery pages URLs that can be parsed for image page URLs which can ultimately be parsed for file URLs and metadata like tags. Boorus fall into this category.</li>
|
||||
<li><h3>Thread Watcher</h3></li>
|
||||
<li>This takes a URL that it will check repeatedly, parsing it for new URLs that it then queues up to be downloaded. It typically stops checking after the 'file velocity' (such as '1 new file per day') drops below a certain level.</li>
|
||||
<li><h3>Single Page Downloader</h3></li>
|
||||
<li>This takes a URL one-time and parses it for more URLs. This is a miscellaneous system for certain simple gallery types. The 'page of images' downloader is one of these.</li>
|
||||
</ul>
|
||||
<p>The system currently supports HTML and JSON parsing.</p>
|
||||
<h3>what does a downloader do?</h3>
|
||||
<p>As an example, in order for hydrus to convert our 'blue_eyes' query into a bunch of files with tags, it needs to:</p>
|
||||
<ul>
|
||||
<li>Present some user interface named 'Safebooru Downloader' to the user that will convert their input of 'blue_eyes' into <a href="https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0">https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0</a>.</li>
|
||||
<li>Recognise <a href="https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0">https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0</a> as a Safebooru Gallery URL.</li>
|
||||
<li>Convert the HTML of a Safebooru Gallery URL into a list URLs like <a href="https://safebooru.org/index.php?page=post&s=view&id=2437965">https://safebooru.org/index.php?page=post&s=view&id=2437965</a> and possibly a 'next page' URL that points to another page of thumbnails.</li>
|
||||
<li>Recognise <a href="https://safebooru.org/index.php?page=post&s=view&id=2437965">https://safebooru.org/index.php?page=post&s=view&id=2437965</a> as a Safebooru Post URL.</li>
|
||||
<li>Convert the HTML of a Safebooru Post URL into a file URL like <a href="https://safebooru.org//images/2329/b6e8c263d691d1c39a2eeba5e00709849d8f864d.jpg">https://safebooru.org//images/2329/b6e8c263d691d1c39a2eeba5e00709849d8f864d.jpg</a> and some tags like: 1girl, bangs, black gloves, blonde hair, blue eyes, braid, closed mouth, day, fingerless gloves, fingernails, gloves, grass, hair ornament, hairclip, hands clasped, creator:hankuri, interlocked fingers, long hair, long sleeves, outdoors, own hands together, parted bangs, pointy ears, character:princess zelda, smile, solo, series:the legend of zelda, underbust.</li>
|
||||
</ul>
|
||||
<p>So we have three components:</p>
|
||||
<ul>
|
||||
<li><b>Downloader:</b> faces the user and converts text input into a series of Gallery URLs.</li>
|
||||
<li><b>URL Class:</b> identifies URLs and informs the client how to deal with them.</li>
|
||||
<li><b>Parser:</b> converts data from URLs into hydrus-understandable metadata.</li>
|
||||
</ul>
|
||||
<p>Thread watchers and single page downloaders do not need the 'Downloader' component, as the input in this case <i>is</i> a URL. You drop an imageboard thread URL on the client and it automatically recognises what it is, launches a thread watcher page for it, and finds the correct parser for the output.</p>
|
||||
<p class="right"><a href="downloader_url_classes.html">Let's learn about URL Classes ----></a></p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
<html>
|
||||
<head>
|
||||
<title>downloader - login manager</title>
|
||||
<link href="hydrus.ico" rel="shortcut icon" />
|
||||
<link href="style.css" rel="stylesheet" type="text/css" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="content">
|
||||
<p><a href="downloader_sharing.html"><---- Back to sharing</a></p>
|
||||
<h3>login</h3>
|
||||
<p class="right"><a href="index.html">Back to the index ----></a></p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -0,0 +1,106 @@
|
|||
<html>
|
||||
<head>
|
||||
<title>downloader - parsers</title>
|
||||
<link href="hydrus.ico" rel="shortcut icon" />
|
||||
<link href="style.css" rel="stylesheet" type="text/css" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="content">
|
||||
<p><a href="downloader_url_classes.html"><---- Back to URL Classes</a></p>
|
||||
<p class="warning">This system still needs work. The user interface remains a hellscape, so I won't put in screenshots for now.</p>
|
||||
<h3>parsers</h3>
|
||||
<p>In hydrus, a parser is an object that takes a single block of HTML or JSON data (as returned by a URL) and returns many kinds of hydrus-level metadata.</p>
|
||||
<p>Parsers are flexible and potentially complicated. You might like to open <i>network->manage parsers</i> and explore the UI as you read this page. Check out how the examples in the client work, and if you want to write a new one, see if there is something already in there that is similar--it is usually easier to duplicate an existing parser and then alter it than to create a new one from scratch every time.</p>
|
||||
<p>There are three main components in the parsing system:</p>
|
||||
<ul>
|
||||
<li><b>Formulae:</b> Take parsable data, search it in some manner, and return 0 to n strings.</li>
|
||||
<li><b>Content Parser:</b> Take parsable data, apply a formula to it to get some strings, and apply a single metadata 'type' and perhaps some additional modifiers.</li>
|
||||
<li><b>Page Parser:</b> Take parsable data, apply content parsers to it, and return all the metadata.</li>
|
||||
</ul>
|
||||
<p>Formulae do the grunt work of parsing and string conversion, content parsers turn the strings into something richer, and page parsers are the containers.</li>
|
||||
<h3>formulae</h3>
|
||||
<p>A formula takes some data and returns some strings. The different kinds are:</p>
|
||||
<ul>
|
||||
<li><h3>html</h3></li>
|
||||
<li>This takes HTML or a sample of HTML--and any regular sort of XML <i>should</i> also work, it is not at all strict--searches for nodes with certain tag names and/or attributes, and then returns those nodes' particular attribute value, string content, or html beneath.</li>
|
||||
<li>The search occurs in steps:</li>
|
||||
<li>(image of a decent formula with several steps)</li>
|
||||
<li>Each step will be applied in turn, starting at the root node and searching beneath the nodes found in the previous step.</li>
|
||||
<li>For instance, if you have this html:</li>
|
||||
<li><pre><html>
|
||||
<body>
|
||||
<div class="media_taglist">
|
||||
<span class="generaltag"><a href="(search page)">blonde hair</a> (3456)</span>
|
||||
<span class="generaltag"><a href="(search page)">blue eyes</a> (4567)</span>
|
||||
<span class="generaltag"><a href="(search page)">bodysuit</a> (5678)</span>
|
||||
<span class="charactertag"><a href="(search page)">samus aran</a> (2345)</span>
|
||||
<span class="artisttag"><a href="(search page)">splashbrush</a> (123)</span>
|
||||
</div>
|
||||
<div class="content">
|
||||
<span class="media">(a whole bunch of content that doesn't have tags in)</span>
|
||||
</div>
|
||||
</body>
|
||||
</html></pre></li>
|
||||
<li>To find the artist, "splashbrush", you would want to:</li>
|
||||
<ul>
|
||||
<li>get every <div> tag with attributes class=media_taglist</li>
|
||||
<li>and then get every <span> tag with attributes class=artisttag</li>
|
||||
<li>and then get the string content of those tags</li>
|
||||
</ul>
|
||||
<li>This will return a single string, "splashbrush". Changing the "artisttag" to "charactertag" or "generaltag" would give you "samus aran" and "blonde hair","blue eyes","bodysuit" respectively.</li>
|
||||
<li>You might be tempted to just go straight for the <span> with class=artisttag, but many sites use the same class to render a sidebar of favourite/popular tags or some other sponsored content, so it is best to make sure you narrow down to the larger <div> container so you don't get anything you don't want.</li>
|
||||
<li>When you add or edit one of these rules, you get this:</li>
|
||||
<li>(image of rule edit panel)</li>
|
||||
<li>Note that you can select to get only the 1st or xth instance of a found tag if you like, which can be useful in situations like this:</li>
|
||||
<li><pre><span class="generaltag">
|
||||
<a href="(add tag)">+</a>
|
||||
<a href="(remove tag)">-</a>
|
||||
<a href="(search page)">blonde hair</a> (3456)
|
||||
</span></pre></li>
|
||||
<li>Without any more attributes, there isn't a good way to distinguish the <a> with "blonde hair" from the other two--so just set 'get the 3rd <a> tag' and you are good.</li>
|
||||
<li>Once you have narrowed down the right nodes you want, you can decide what to return. So, given a node of:</li>
|
||||
<li><pre><a href="(URL A)" class="thumb">Forest Glade</a></pre></li>
|
||||
<li>Returning the 'href' attribute would return the string "(URL A)", returning the string content would give "Forest Glade", and returning the full html would give "<a href="(URL A)" class="thumb">Forest Glade</a>". This last choice is useful in complicated situations where you want a second, separated layer of parsing, which we will get to later.</li>
|
||||
<li><h3>json</h3></li>
|
||||
<li>This takes some JSON and does a similar style of search:</li>
|
||||
<li>(image of edit formula panel)</li>
|
||||
<li>It is a bit simpler than HTML--if the current node is a list (called an 'Array' in JSON), you can fetch every item or the xth item, and if it is a dictionary (called an 'Object' in JSON), you can fetch a particular string entry. Since you can't jump down several layers with attribute lookups or tag names, you have to go down every layer one at a time. In any case, if you have something like this:</li>
|
||||
<li><a href="json_thread_example.png"><img src="json_thread_example.png" width="50%" height="50%"/></a></li>
|
||||
<li>Then searching for "posts"->1st list item->"sub" will give you "Nobody like kino here.".</li>
|
||||
<li>Then searching for "posts"->all list items->"tim" will give you the three file hashes (since the third post has no file attached, the parser skips over it without complaint).</li>
|
||||
<li>Searching for "posts"->1st list item->"com" will give you the OP's comment, <span class="dealwithit">~AS RAW UNPARSED HTML~</span>.</li>
|
||||
<li>The default is to fetch the final nodes' 'data content', which means coercing simple variables into strings. If the current node is a list or dict, no string is returned.</li>
|
||||
<li>But if you like, you can return the json beneath the current node (which, like HTML, includes the current node). This again will come in useful later.</li>
|
||||
<li><h3>compound</h3></li>
|
||||
<li>If you want to create a string from multiple parsed strings--for instance by appending the 'tim' and the 'ext' in our json example together--you can use a Compound formula. This fetches multiple lists of strings and tries to place them into a single string using \1 \2 \3 regex substitution syntax:</li>
|
||||
<li>(image of the edit panel--use the thread watcher one with complicated gubbins)</li>
|
||||
<li>This is where the magic happens, sometimes, so keep it in mind if you need to do something cleverer than the data you have seems to provide.</li>
|
||||
<li><h3>context variable</h3></li>
|
||||
<li>desc</li>
|
||||
<li>ui walkthrough</li>
|
||||
<li>misc</li>
|
||||
</ul>
|
||||
<p>talk about string match and string converter</p>
|
||||
<p>how to test</p>
|
||||
<p>It is a great idea to check the html or json you are trying to parse with your browser. Most web browsers have great developer tools that let you walk through the different nodes in a pretty way. The JSON image above is one of the views Firefox provides if you simply enter a JSON URL.</p>
|
||||
<h3>content parser</h3>
|
||||
<p>different types and what they mean</p>
|
||||
<p>hash needs conversion to bytes</p>
|
||||
<p>vetos</p>
|
||||
<h3>page parser</h3>
|
||||
<p>pre-parsing conversion example for tumblr</p>
|
||||
<p>example urls are helpful</p>
|
||||
<p>mention vetos again</p>
|
||||
<p>subsidiary page parsers and what that is for</p>
|
||||
<h3>page example</h3>
|
||||
<p>do a danbooru example with sample image stuff</p>
|
||||
<h3>gallery example</h3>
|
||||
<p>something with an API?</p>
|
||||
<h3>thread example</h3>
|
||||
<p>subsidiary page parsers in the example</p>
|
||||
<p>source time and subject->comment fallback fun</p>
|
||||
<p>The context variable bit to fetch the right board for the file url</p>
|
||||
<p class="right"><a href="downloader_downloaders.html">Let's learn about Downloaders ----></a></p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
<html>
|
||||
<head>
|
||||
<title>downloader - sharing</title>
|
||||
<link href="hydrus.ico" rel="shortcut icon" />
|
||||
<link href="style.css" rel="stylesheet" type="text/css" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="content">
|
||||
<p><a href="downloader_completion.html"><---- Back to putting downloaders together</a></p>
|
||||
<h3>sharing</h3>
|
||||
<p class="right"><a href="downloader_login.html">Onto the login manager ----></a></p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -0,0 +1,128 @@
|
|||
<html>
|
||||
<head>
|
||||
<title>downloader - url classes</title>
|
||||
<link href="hydrus.ico" rel="shortcut icon" />
|
||||
<link href="style.css" rel="stylesheet" type="text/css" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="content">
|
||||
<p><a href="downloader_intro.html"><---- Back to the introduction</a></p>
|
||||
<h3>url classes</h3>
|
||||
<p>The fundamental connective part of the downloader system is the 'URL Class'. This object identifies and normalises URLs and links them to other components. When the client handles or presents a URL, it consults the respective URL Class on what to do.</p>
|
||||
<h3>the types of url</h3>
|
||||
<p>For hydrus, an URL is useful if it is one of:</p>
|
||||
<ul>
|
||||
<li><h3>File URL</h3></li>
|
||||
<li>
|
||||
<p>This returns the full, raw media file with no HTML wrapper. They typically end in a filename like <a href="http://safebooru.org//images/2333/cab1516a7eecf13c462615120ecf781116265f17.jpg">http://safebooru.org//images/2333/cab1516a7eecf13c462615120ecf781116265f17.jpg</a>, but sometimes they have a more complicated fetch command ending like 'file.php?id=123456' or '/post/content/123456'.</p>
|
||||
<p>These URLs are remembered for the file in the 'known urls' list, so if the client happens to encounter the same URL in future, it can determine whether it can skip the download because the file is already in the database or has previously been deleted.</p>
|
||||
<p>It is not important that File URLs are matched. The client does not need to match a File URL in order to download it (and will typically assume unmatched URLs <i>are</i> File URLs), but you might want to particularly specify them if you discover File URLs are being confused for Post URLs or something.</p>
|
||||
</li>
|
||||
<li><h3>Post URL</h3></li>
|
||||
<li>
|
||||
<p>This typically contains one File URL and some metadata like tags. They sometimes present multiple sizes (like 'sample' vs 'full size') of the file or even different formats (like 'ugoira' vs 'webm'). The Post URL for the file above, <a href="http://safebooru.org/index.php?page=post&s=view&id=2429668">http://safebooru.org/index.php?page=post&s=view&id=2429668</a> has this 'sample' presentation. Finding the best File URL in these cases can be tricky!</p>
|
||||
<p>This URL is also saved to 'known urls' and can be skipped if it has previously been downloaded. It will also appear in the media viewer as a clickable link. Since the user may want to load these in their browser, it is important that they stay intact and valid.</p>
|
||||
</li>
|
||||
<li><h3>Gallery URL</h3></li>
|
||||
<li>This presents a list of Post URLs or File URLs. They often also present a 'next page' URL and sometimes say when the files it links were posted, which can be used to calculate how fast files are being posted. It could be a page like <a href="http://safebooru.org/index.php?page=post&s=list&tags=yorha_no._2_type_b&pid=0">http://safebooru.org/index.php?page=post&s=list&tags=yorha_no._2_type_b&pid=0</a> or an API URL like <a href="http://safebooru.org/index.php?page=dapi&s=post&tags=yorha_no._2_type_b&q=index&pid=0">http://safebooru.org/index.php?page=dapi&s=post&tags=yorha_no._2_type_b&q=index&pid=0</a>.</li>
|
||||
<li><h3>Watchable URL</h3></li>
|
||||
<li>This is the same as a Gallery URL but represents an ephemeral page that receives new files much faster than a gallery but will soon 'die' and be deleted. For our purposes, this typically means imageboard threads.</li>
|
||||
</ul>
|
||||
<h3>the components of a url</h3>
|
||||
<p>For our purposes, a URL string has four parts:</p>
|
||||
<ul>
|
||||
<li><b>Scheme:</b> "http" or "https"</li>
|
||||
<li><b>Location/Domain:</b> "safebooru.org" or "i.4cdn.org" or "cdn002.somebooru.net"</li>
|
||||
<li><b>Path Components:</b> "index.php" or "tesla/res/7518.json" or "pictures/user/daruak/page/2" or "art/Commission-animation-Elsa-and-Anna-541820782"</li>
|
||||
<li><b>Query Parameters:</b> "page=post&s=list&tags=yorha_no._2_type_b&pid=40" or "page=post&s=view&id=2429668"</li>
|
||||
</ul>
|
||||
<p>So, let's look at the 'edit url class' panel, which is found under <i>network->manage url classes</i>:</p>
|
||||
<p><img src="downloader_edit_url_class_panel.png" /></p>
|
||||
<p>A TBIB File Page like <a href="https://tbib.org/index.php?page=post&s=view&id=6391256">https://tbib.org/index.php?page=post&s=view&id=6391256</a> is a Post URL. Let's go over the four components again:</p>
|
||||
<ul>
|
||||
<li><h3>Scheme</h3></li>
|
||||
<li>
|
||||
<p>TBIB supports http and https, so I have set the 'preferred' scheme to https. Any 'http' TBIB URL a user inputs will be automatically converted to https.</p>
|
||||
</li>
|
||||
<li><h3>Location/Domain</h3></li>
|
||||
<li>
|
||||
<p>For File URLs, the domain is always "tbib.org".</p>
|
||||
<p>The 'allow' and 'keep' subdomains checkboxes let you determine if a URL with "rule34.booru.org" will match a URL Class with "booru.org" domain and if that subdomain should be remembered going forward. In the booru.org case, the subdomain is very important, and removing it breaks the URL, but in cases where a site farms out File URLs to CDN servers on subdomains (like randomly serving a mirror of "https://muhbooru.org/file/123456" on "https://srv2.muhbooru.org/file/123456") and removing the subdomain still gives a valid URL, you do not wish to keep the subdomain. As it is, TBIB does not use subdomains, so these options do not matter in this case.</p>
|
||||
<p>I am not totally happy with how allow and keep subdomains work, so I may replace this in future.</p>
|
||||
</li>
|
||||
<li><h3>Path Components</h3></li>
|
||||
<li>
|
||||
<p>TBIB just uses a single "index.php" on the root directory, so the path is not complicated. Were it longer (like "gallery/cgi/index.php", we would add more ("gallery" and "cgi"), and since the path of a URL has a strict order, we would need to arrange the items in the listbox there so they were sorted correctly.</p>
|
||||
</li>
|
||||
<li><h3>Query Parameters</h3></li>
|
||||
<li>
|
||||
<p>TBIB's index.php takes many query parameters to render different page types. Note that the Post URL uses "s=view", while TBIB Gallery URLs use "s=list". In any case, for a Post URL, "id", "page", and "s" are necessary and sufficient.</p>
|
||||
</li>
|
||||
</ul>
|
||||
<p>This URL Class will be assigned to any URL that matches the location, path, and query. Missing components in the URL will invalidate the match but additonal components will not!</p>
|
||||
<p>For instance:</p>
|
||||
<ul>
|
||||
<li>URL A: https://8ch.net/tv/res/1002432.html</li>
|
||||
<li>URL B: https://8ch.net/tv/res</li>
|
||||
<li>URL C: https://8ch.net/tv/res/1002432</li>
|
||||
<li>URL D: https://8ch.net/tv/res/1002432.json</li>
|
||||
<li>URL Class that looks for "(characters)/res/(numbers).html" for the path</li>
|
||||
</ul>
|
||||
<p>Only URL A will match</p>
|
||||
<p>And:</p>
|
||||
<ul>
|
||||
<li>URL A: https://boards.4chan.org/m/thread/16086187</li>
|
||||
<li>URL B: https://boards.4chan.org/m/thread/16086187/ssg-super-sentai-general-651</li>
|
||||
<li>URL Class that looks for "(characters)/thread/(numbers)" for the path</li>
|
||||
</ul>
|
||||
<p>Both URL A and B will match</p>
|
||||
<p>And:</p>
|
||||
<ul>
|
||||
<li>URL A: https://www.pixiv.net/member_illust.php?mode=medium&illust_id=66476204</li>
|
||||
<li>URL B: https://www.pixiv.net/member_illust.php?mode=medium&illust_id=66476204&lang=jp</li>
|
||||
<li>URL C: https://www.pixiv.net/member_illust.php?mode=medium</li>
|
||||
<li>URL Class that looks for "illust_id=(numbers)" in the query</li>
|
||||
</ul>
|
||||
<p>Both URL A and B will match, URL C will not</p>
|
||||
<p>If multiple URL Classes match a URL, the client will try to assign the most 'complicated' one, with the most path components and then query parameters.</p>
|
||||
<p>Given two example URLs and URL Classes:</p>
|
||||
<ul>
|
||||
<li>URL A: https://somebooru.com/post/123456</li>
|
||||
<li>URL B: https://somebooru.com/post/123456/manga_subpage/2</li>
|
||||
<li>URL Class A that looks for "post/(number)" for the path</li>
|
||||
<li>URL Class B that looks for "post/(number)/manga_subpage/(number)" for the path</li>
|
||||
</ul>
|
||||
<p>URL A will match URL Class A but not URL Class B and so will receive A.</p>
|
||||
<p>URL B will match both and will receive URL Class B as it is more complicated.</p>
|
||||
<p>This situation is not common, but I expect it to be an issue with Pixiv, where some Post URLs link to a subset of manga pages that have their own gallery system, wew.</p>
|
||||
<h3>string matches</h3>
|
||||
<p>As you edit these components, you will be presented with the Edit String Match Panel:</p>
|
||||
<p><img src="edit_string_match_panel.png" /></p>
|
||||
<p>This lets you set the type of string that will be valid for that component. If a given path or query component does not match the rules given here, the URL will not match the URL Class. Most of the time you will probably want to set 'fixed characters' of something like "post" or "index.php", but if the component you are editing is more complicated and could have a range of different valid values, you can specify just numbers or letters or even a regex pattern. If you try to do something complicated, experiment with the 'example string' entry to make sure you have it set how you think.</p>
|
||||
<p>Don't go overboard with this stuff, though--most sites do not have super-fine distinctions between their different URL types, and hydrus users will not be dropping user account or logout pages or whatever on the client, so you can be fairly liberal with the rules.</p>
|
||||
<h3>normalising urls</h3>
|
||||
<p>Different URLs can give the same page. The http and https versions of a URL are typically the same, and "http://site.com/index.php?s=post&id=123456" results in the same content as "http://site.com/index.php?id=123456&s=post", and "https://e621.net/post/show/1421754/abstract_background-animal_humanoid-blush-brown_ey" is the same as "https://e621.net/post/show/1421754".</p>
|
||||
<p>Since we are in the business of storing and comparing URLs, we want to 'normalise' them to a single comparable beautiful value. You see a preview of this normalisation on the edit panel.</p>
|
||||
<p>Gallery and Watchable URLs are not compared, so a normalise call for them only switches their http/https to the preferred value, but File and Post URLs will cut out any surplus path or query components and will alphabetise the query arguments as well.</p>
|
||||
<p>Since File and Post URLs will remove anything surplus, be careful that you not leave out anything important in your rules. Make sure what you have is both necessary (nothing can be removed and still keep it valid) and sufficient (no more needs to be added to make it valid). It is a good idea to try pasting the 'normalised' version of the example URL into your browser, just to check it still works.</p>
|
||||
<h3>gallery rules do not need to be sufficient</h3>
|
||||
<p class="warning">Advanced--feel free to skip for now</p>
|
||||
<p>For Gallery URLs, however, it can sometimes be useful to specify just a set of necessary rules. This saves your time and covers a broader set of URLs like these:</p>
|
||||
<ul>
|
||||
<li>https://www.hentai-foundry.com/pictures/user/Sparrow/scraps/page/3</li>
|
||||
<li>https://www.hentai-foundry.com/pictures/user/Sparrow/scraps/page/1</li>
|
||||
<li>https://www.hentai-foundry.com/pictures/user/Sparrow/scraps (note that this gives the same result as the /page/1 URL)</li>
|
||||
</ul>
|
||||
<p>Rather than making two rules--one with the additional "/page/(number)" and one without--you can just make one for "pictures/user/(characters)/scraps", which will match all three examples above.</p>
|
||||
<p>While hydrus downloaders tend to generate valid first page URLs with something like "/page/1" or "pid=0" or "index=0", the sites themselves tend to link a 'bare' URL to a user browsing with a mouse. If you demand the 'page' or 'index' part in your Gallery URL Classes, a user who finds a nice gallery and tries to drop the first page's URL, as the site presented it, onto the client will only get a 'Couldn't find a URL Class for that!' error.</p>
|
||||
<p>But if there isn't a nice way to create a single non-ambiguous class, just make multiple.</p>
|
||||
<h3>api urls</h3>
|
||||
<p>If you know that a URL has an API backend, you can tell the client to use that API URL when it fetches data. The API URL needs its own URL Class.</p>
|
||||
<p>To define the relationship, click the "String Converter" button, which gives you this:</p>
|
||||
<p><img src="edit_string_converter_panel.png" /></p>
|
||||
<p>You may have seen this panel elsewhere. It lets you convert a string to another over a number of transformation steps. The steps can be as simple as adding or removing some characters or applying a full regex substitution. For API URLs, you are mostly looking to isolate some unique identifying data ("m/thread/16086187" in this case) and then substituting that into the new API path. It is worth testing this with several different examples!</p>
|
||||
<p>When the client links regular URLs to API URLs like this, it will still associate the human-pretty regular URL when it needs to display to the user and record 'known urls' and so on. The API is just a quick lookup when it actually fetches and parses the respective data.</p>
|
||||
<p class="right"><a href="downloader_parsers.html">Let's learn about Parsers ----></a></p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
|
|
@ -48,6 +48,16 @@
|
|||
<li><a href="wine.html">running a client or server in wine</a></li>
|
||||
<li><a href="running_from_source.html">running a client or server from source</a></li>
|
||||
</ul>
|
||||
<li><h3>making a downloader (under construction)</h3></li>
|
||||
<ul>
|
||||
<li><a href="downloader_intro.html">introduction</a></li>
|
||||
<li><a href="downloader_url_classes.html">url classes</a></li>
|
||||
<li><a href="downloader_parsers.html">parsers</a></li>
|
||||
<li><a href="downloader_downloaders.html">downloaders</a></li>
|
||||
<li><a href="downloader_completion.html">putting it all together</a></li>
|
||||
<li><a href="downloader_sharing.html">sharing downloaders</a></li>
|
||||
<li><a href="downloader_login.html">login manager</a></li>
|
||||
</ul>
|
||||
<li><h3>misc</h3></li>
|
||||
<ul>
|
||||
<li><a href="privacy.html">privacy</a></li>
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 116 KiB |
|
|
@ -11,6 +11,7 @@ import HydrusFileHandling
|
|||
import HydrusPaths
|
||||
import HydrusSerialisable
|
||||
import HydrusSessions
|
||||
import HydrusThreading
|
||||
import itertools
|
||||
import json
|
||||
import os
|
||||
|
|
@ -2017,7 +2018,7 @@ class ThumbnailCache( object ):
|
|||
|
||||
last_paused = HydrusData.GetNowPrecise()
|
||||
|
||||
while not HG.view_shutdown:
|
||||
while not HydrusThreading.IsThreadShuttingDown():
|
||||
|
||||
with self._lock:
|
||||
|
||||
|
|
|
|||
|
|
@ -167,6 +167,28 @@ class Controller( HydrusController.HydrusController ):
|
|||
raise HydrusExceptions.ShutdownException()
|
||||
|
||||
|
||||
def CallLaterWXSafe( self, window, delay, func, *args, **kwargs ):
|
||||
|
||||
call = HydrusData.Call( func, *args, **kwargs )
|
||||
|
||||
job = ClientThreading.WXAwareJob( self, self._job_scheduler, window, call, initial_delay = delay )
|
||||
|
||||
self._job_scheduler.AddJob( job )
|
||||
|
||||
return job
|
||||
|
||||
|
||||
def CallRepeatingWXSafe( self, window, period, delay, func, *args, **kwargs ):
|
||||
|
||||
call = HydrusData.Call( func, *args, **kwargs )
|
||||
|
||||
job = ClientThreading.WXAwareRepeatingJob( self, self._job_scheduler, window, call, period, initial_delay = delay )
|
||||
|
||||
self._job_scheduler.AddJob( job )
|
||||
|
||||
return job
|
||||
|
||||
|
||||
def CheckAlreadyRunning( self ):
|
||||
|
||||
while HydrusData.IsAlreadyRunning( self.db_dir, 'client' ):
|
||||
|
|
@ -370,14 +392,19 @@ class Controller( HydrusController.HydrusController ):
|
|||
|
||||
with ClientGUIDialogs.DialogYesNo( self._splash, text, title = 'Maintenance is due' ) as dlg_yn:
|
||||
|
||||
call_later = ClientThreading.CallLater( dlg_yn, 15, dlg_yn.EndModal, wx.ID_NO )
|
||||
job = self.CallLaterWXSafe( dlg_yn, 15, dlg_yn.EndModal, wx.ID_NO )
|
||||
|
||||
if dlg_yn.ShowModal() == wx.ID_YES:
|
||||
try:
|
||||
|
||||
HG.do_idle_shutdown_work = True
|
||||
if dlg_yn.ShowModal() == wx.ID_YES:
|
||||
|
||||
HG.do_idle_shutdown_work = True
|
||||
|
||||
|
||||
finally:
|
||||
|
||||
job.Cancel()
|
||||
|
||||
|
||||
call_later.Stop()
|
||||
|
||||
|
||||
else:
|
||||
|
|
|
|||
|
|
@ -3190,6 +3190,17 @@ class DB( HydrusDB.HydrusDB ):
|
|||
|
||||
|
||||
|
||||
def _DisplayCatastrophicError( self, text ):
|
||||
|
||||
message = 'The db encountered a serious error! This is going to be written to the log as well, but here it is for a screenshot:'
|
||||
message += os.linesep * 2
|
||||
message += text
|
||||
|
||||
HydrusData.DebugPrint( message )
|
||||
|
||||
wx.SafeShowMessage( message )
|
||||
|
||||
|
||||
def _ExportToTagArchive( self, path, service_key, hash_type, hashes = None ):
|
||||
|
||||
# This could nicely take a whitelist or a blacklist for namespace filtering
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ def DAEMONCheckImportFolders( controller ):
|
|||
|
||||
import_folder = controller.Read( 'serialisable_named', HydrusSerialisable.SERIALISABLE_TYPE_IMPORT_FOLDER, name )
|
||||
|
||||
if controller.options[ 'pause_import_folders_sync' ] or HG.view_shutdown:
|
||||
if controller.options[ 'pause_import_folders_sync' ] or HydrusThreading.IsThreadShuttingDown():
|
||||
|
||||
break
|
||||
|
||||
|
|
|
|||
|
|
@ -127,6 +127,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
|
|||
self._controller.sub( self, 'NotifyNewServices', 'notify_new_services_gui' )
|
||||
self._controller.sub( self, 'NotifyNewSessions', 'notify_new_sessions' )
|
||||
self._controller.sub( self, 'NotifyNewUndo', 'notify_new_undo' )
|
||||
self._controller.sub( self, 'PresentImportedFilesToPage', 'imported_files_to_page' )
|
||||
self._controller.sub( self, 'RenamePage', 'rename_page' )
|
||||
self._controller.sub( self, 'SetDBLockedStatus', 'db_locked_status' )
|
||||
self._controller.sub( self, 'SetMediaFocus', 'set_media_focus' )
|
||||
|
|
@ -141,7 +142,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
|
|||
|
||||
self.SetSizer( vbox )
|
||||
|
||||
ClientGUITopLevelWindows.SetTLWSizeAndPosition( self, self._frame_key )
|
||||
ClientGUITopLevelWindows.SetInitialTLWSizeAndPosition( self, self._frame_key )
|
||||
|
||||
self.Show( True )
|
||||
|
||||
|
|
@ -555,7 +556,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
|
|||
self._notebook.SaveGUISession( 'last session' )
|
||||
|
||||
# session save causes a db read in the menu refresh, so let's put this off just a bit
|
||||
ClientThreading.CallLater( self, 1.5, self._controller.Write, 'backup', path )
|
||||
self._controller.CallLater( 1.5, self._controller.Write, 'backup', path )
|
||||
|
||||
|
||||
|
||||
|
|
@ -760,8 +761,9 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
|
|||
|
||||
self._controller.pub( 'message', job_key )
|
||||
|
||||
ClientThreading.CallLater( self, 2, job_key.SetVariable, 'popup_text_2', 'Pulsing subjob' )
|
||||
ClientThreading.CallLater( self, 2, job_key.SetVariable, 'popup_gauge_2', ( 0, None ) )
|
||||
self._controller.CallLater( 2.0, job_key.SetVariable, 'popup_text_2', 'Pulsing subjob' )
|
||||
|
||||
self._controller.CallLater( 2.0, job_key.SetVariable, 'popup_gauge_2', ( 0, None ) )
|
||||
|
||||
#
|
||||
|
||||
|
|
@ -773,7 +775,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
|
|||
|
||||
for i in range( 1, 4 ):
|
||||
|
||||
ClientThreading.CallLater( self, 0.5 * i, HydrusData.ShowText, 'This is a delayed popup message -- ' + str( i ) )
|
||||
self._controller.CallLater( 0.5 * i, HydrusData.ShowText, 'This is a delayed popup message -- ' + str( i ) )
|
||||
|
||||
|
||||
|
||||
|
|
@ -1636,7 +1638,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
|
|||
ClientGUIMenus.AppendMenu( debug, report_modes, 'report modes' )
|
||||
|
||||
ClientGUIMenus.AppendMenuItem( self, debug, 'make some popups', 'Throw some varied popups at the message manager, just to check it is working.', self._DebugMakeSomePopups )
|
||||
ClientGUIMenus.AppendMenuItem( self, debug, 'make a popup in five seconds', 'Throw a delayed popup at the message manager, giving you time to minimise or otherwise alter the client before it arrives.', ClientThreading.CallLater, self, 5, HydrusData.ShowText, 'This is a delayed popup message.' )
|
||||
ClientGUIMenus.AppendMenuItem( self, debug, 'make a popup in five seconds', 'Throw a delayed popup at the message manager, giving you time to minimise or otherwise alter the client before it arrives.', self._controller.CallLater, 5, HydrusData.ShowText, 'This is a delayed popup message.' )
|
||||
ClientGUIMenus.AppendMenuItem( self, debug, 'force a gui layout now', 'Tell the gui to relayout--useful to test some gui bootup layout issues.', self.Layout )
|
||||
ClientGUIMenus.AppendMenuItem( self, debug, 'flush log', 'Command the log to write any buffered contents to hard drive.', HydrusData.DebugPrint, 'Flushing log' )
|
||||
ClientGUIMenus.AppendMenuItem( self, debug, 'print garbage', 'Print some information about the python garbage to the log.', self._DebugPrintGarbage )
|
||||
|
|
@ -1863,7 +1865,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
|
|||
|
||||
last_session_save_period_minutes = self._controller.new_options.GetInteger( 'last_session_save_period_minutes' )
|
||||
|
||||
ClientThreading.CallLater( self, last_session_save_period_minutes * 60, self.SaveLastSession )
|
||||
self._controller.CallLaterWXSafe( self, last_session_save_period_minutes * 60, self.SaveLastSession )
|
||||
|
||||
|
||||
def _ManageAccountTypes( self, service_key ):
|
||||
|
|
@ -3174,7 +3176,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
|
|||
|
||||
if self.IsIconized():
|
||||
|
||||
ClientThreading.CallLater( self, 10, self.AddModalMessage, job_key )
|
||||
self._controller.CallLaterWXSafe( self, 10, self.AddModalMessage, job_key )
|
||||
|
||||
else:
|
||||
|
||||
|
|
@ -3460,16 +3462,19 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
|
|||
|
||||
with ClientGUIDialogs.DialogYesNo( self, text ) as dlg:
|
||||
|
||||
timer = ClientThreading.CallLater( self, 15, dlg.EndModal, wx.ID_YES )
|
||||
job = self._controller.CallLaterWXSafe( dlg, 15, dlg.EndModal, wx.ID_YES )
|
||||
|
||||
if dlg.ShowModal() == wx.ID_NO:
|
||||
try:
|
||||
|
||||
timer.Stop()
|
||||
if dlg.ShowModal() == wx.ID_NO:
|
||||
|
||||
return False
|
||||
|
||||
|
||||
return False
|
||||
finally:
|
||||
|
||||
job.Cancel()
|
||||
|
||||
|
||||
timer.Stop()
|
||||
|
||||
|
||||
|
||||
|
|
@ -3533,7 +3538,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
|
|||
|
||||
else:
|
||||
|
||||
ClientThreading.CallLater( self, 2, self.Destroy )
|
||||
self._controller.CallLaterWXSafe( self, 2, self.Destroy )
|
||||
|
||||
self._controller.CreateSplash()
|
||||
|
||||
|
|
@ -3825,6 +3830,11 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
|
|||
return False
|
||||
|
||||
|
||||
def PresentImportedFilesToPage( self, hashes, page_name ):
|
||||
|
||||
dest_page = self._notebook.PresentImportedFilesToPage( hashes, page_name )
|
||||
|
||||
|
||||
def RefreshMenu( self ):
|
||||
|
||||
if not self:
|
||||
|
|
@ -3836,7 +3846,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
|
|||
|
||||
if db_going_to_hang_if_we_hit_it:
|
||||
|
||||
ClientThreading.CallLater( self, 2.5, self.RefreshMenu )
|
||||
self._controller.CallLaterWXSafe( self, 0.5, self.RefreshMenu )
|
||||
|
||||
return
|
||||
|
||||
|
|
@ -3934,7 +3944,7 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
|
|||
|
||||
last_session_save_period_minutes = self._controller.new_options.GetInteger( 'last_session_save_period_minutes' )
|
||||
|
||||
ClientThreading.CallLater( self, last_session_save_period_minutes * 60, self.SaveLastSession )
|
||||
self._controller.CallLaterWXSafe( self, last_session_save_period_minutes * 60, self.SaveLastSession )
|
||||
|
||||
|
||||
def SetMediaFocus( self ):
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import ClientGUIMenus
|
|||
import ClientSearch
|
||||
import collections
|
||||
import HydrusConstants as HC
|
||||
import HydrusData
|
||||
import HydrusExceptions
|
||||
import HydrusGlobals as HG
|
||||
import HydrusTags
|
||||
|
|
@ -120,19 +121,13 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
|
||||
self._initial_matches_fetched = False
|
||||
|
||||
self._move_hide_timer = None
|
||||
self._move_hide_job = None
|
||||
|
||||
if self._float_mode:
|
||||
|
||||
self.Bind( wx.EVT_MOVE, self.EventMove )
|
||||
self.Bind( wx.EVT_SIZE, self.EventMove )
|
||||
|
||||
self.Bind( wx.EVT_TIMER, self.TIMEREventDropdownHide, id = ID_TIMER_DROPDOWN_HIDE )
|
||||
|
||||
self._move_hide_timer = wx.Timer( self, id = ID_TIMER_DROPDOWN_HIDE )
|
||||
|
||||
self._move_hide_timer.Start( 1, wx.TIMER_ONE_SHOT )
|
||||
|
||||
tlp.Bind( wx.EVT_MOVE, self.EventMove )
|
||||
|
||||
parent = self
|
||||
|
|
@ -155,13 +150,11 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
|
||||
|
||||
|
||||
self.Bind( wx.EVT_TIMER, self.TIMEREventLag, id = ID_TIMER_AC_LAG )
|
||||
|
||||
HG.client_controller.sub( self, '_UpdateBackgroundColour', 'notify_new_colourset' )
|
||||
|
||||
self._lag_timer = wx.Timer( self, id = ID_TIMER_AC_LAG )
|
||||
self._refresh_list_job = None
|
||||
|
||||
wx.CallAfter( self._UpdateList )
|
||||
self._ScheduleListRefresh( 0.0 )
|
||||
|
||||
|
||||
def _BroadcastChoices( self, predicates ):
|
||||
|
|
@ -176,6 +169,14 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
self._BroadcastChoices( { text } )
|
||||
|
||||
|
||||
def _CancelScheduledListRefresh( self ):
|
||||
|
||||
if self._refresh_list_job is not None:
|
||||
|
||||
self._refresh_list_job.Cancel()
|
||||
|
||||
|
||||
|
||||
def _GenerateMatches( self ):
|
||||
|
||||
raise NotImplementedError()
|
||||
|
|
@ -196,6 +197,27 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
raise NotImplementedError()
|
||||
|
||||
|
||||
def _ScheduleListRefresh( self, delay ):
|
||||
|
||||
if self._refresh_list_job is not None and delay == 0.0:
|
||||
|
||||
self._refresh_list_job.MoveNextWorkTimeToNow()
|
||||
|
||||
else:
|
||||
|
||||
self._CancelScheduledListRefresh()
|
||||
|
||||
self._refresh_list_job = HG.client_controller.CallLaterWXSafe( self, delay, self._UpdateList )
|
||||
|
||||
|
||||
|
||||
def _SetListDirty( self ):
|
||||
|
||||
self._cache_text = None
|
||||
|
||||
self._ScheduleListRefresh( 0.0 )
|
||||
|
||||
|
||||
def _ShouldShow( self ):
|
||||
|
||||
tlp_active = self.GetTopLevelParent().IsActive() or self._dropdown_window.IsActive()
|
||||
|
|
@ -309,18 +331,38 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
self._BroadcastChoices( predicates )
|
||||
|
||||
|
||||
def CleanBeforeDestroy( self ):
|
||||
def DropdownHideShow( self ):
|
||||
|
||||
if self._move_hide_timer is not None:
|
||||
try:
|
||||
|
||||
self._move_hide_timer.Stop()
|
||||
self._move_hide_timer = None
|
||||
should_show = self._ShouldShow()
|
||||
|
||||
|
||||
if self._lag_timer is not None:
|
||||
if should_show:
|
||||
|
||||
self._ShowDropdown()
|
||||
|
||||
if self._move_hide_job is not None:
|
||||
|
||||
self._move_hide_job.Cancel()
|
||||
|
||||
self._move_hide_job = None
|
||||
|
||||
|
||||
else:
|
||||
|
||||
self._HideDropdown()
|
||||
|
||||
|
||||
self._lag_timer.Stop()
|
||||
self._lag_timer = None
|
||||
except:
|
||||
|
||||
if self._move_hide_job is not None:
|
||||
|
||||
self._move_hide_job.Cancel()
|
||||
|
||||
self._move_hide_job = None
|
||||
|
||||
|
||||
raise
|
||||
|
||||
|
||||
|
||||
|
|
@ -338,9 +380,7 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
|
||||
elif key == wx.WXK_SPACE and event.RawControlDown(): # this is control, not command on os x, for which command+space does some os stuff
|
||||
|
||||
self._UpdateList()
|
||||
|
||||
self._lag_timer.Stop()
|
||||
self._ScheduleListRefresh( 0.0 )
|
||||
|
||||
elif self._intercept_key_events:
|
||||
|
||||
|
|
@ -397,9 +437,9 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
|
||||
def EventKillFocus( self, event ):
|
||||
|
||||
if self._move_hide_timer:
|
||||
if self._float_mode:
|
||||
|
||||
self._move_hide_timer.Start( 1, wx.TIMER_ONE_SHOT )
|
||||
self.DropdownHideShow()
|
||||
|
||||
|
||||
event.Skip()
|
||||
|
|
@ -453,11 +493,19 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
|
||||
def EventMove( self, event ):
|
||||
|
||||
self._HideDropdown()
|
||||
|
||||
if self._move_hide_timer:
|
||||
if self._float_mode:
|
||||
|
||||
self._move_hide_timer.Start( 250, wx.TIMER_ONE_SHOT )
|
||||
self._HideDropdown()
|
||||
|
||||
if self._ShouldShow():
|
||||
|
||||
if self._move_hide_job is None:
|
||||
|
||||
self._move_hide_job = HG.client_controller.CallRepeatingWXSafe( self._dropdown_window, 0.25, 0.0, self.DropdownHideShow )
|
||||
|
||||
|
||||
self._move_hide_job.Delay( 0.25 )
|
||||
|
||||
|
||||
|
||||
event.Skip()
|
||||
|
|
@ -465,9 +513,9 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
|
||||
def EventSetFocus( self, event ):
|
||||
|
||||
if self._move_hide_timer:
|
||||
if self._float_mode:
|
||||
|
||||
self._move_hide_timer.Start( 1, wx.TIMER_ONE_SHOT )
|
||||
self.DropdownHideShow()
|
||||
|
||||
|
||||
event.Skip()
|
||||
|
|
@ -479,7 +527,7 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
|
||||
if num_chars == 0:
|
||||
|
||||
self._UpdateList()
|
||||
self._ScheduleListRefresh( 0.0 )
|
||||
|
||||
elif HC.options[ 'fetch_ac_results_automatically' ]:
|
||||
|
||||
|
|
@ -487,70 +535,21 @@ class AutoCompleteDropdown( wx.Panel ):
|
|||
|
||||
self._next_updatelist_is_probably_fast = self._next_updatelist_is_probably_fast and num_chars > len( self._last_search_text )
|
||||
|
||||
if self._next_updatelist_is_probably_fast: self._UpdateList()
|
||||
if self._next_updatelist_is_probably_fast:
|
||||
|
||||
self._ScheduleListRefresh( 0.0 )
|
||||
|
||||
elif num_chars < char_limit:
|
||||
|
||||
self._lag_timer.Start( long_wait, wx.TIMER_ONE_SHOT )
|
||||
self._ScheduleListRefresh( long_wait / 1000.0 )
|
||||
|
||||
else:
|
||||
|
||||
self._lag_timer.Start( short_wait, wx.TIMER_ONE_SHOT )
|
||||
self._ScheduleListRefresh( short_wait / 1000.0 )
|
||||
|
||||
|
||||
|
||||
|
||||
def RefreshList( self ):
|
||||
|
||||
self._cache_text = None
|
||||
|
||||
self._UpdateList()
|
||||
|
||||
|
||||
def TIMEREventDropdownHide( self, event ):
|
||||
|
||||
try:
|
||||
|
||||
should_show = self._ShouldShow()
|
||||
|
||||
if should_show:
|
||||
|
||||
self._ShowDropdown()
|
||||
|
||||
else:
|
||||
|
||||
self._HideDropdown()
|
||||
|
||||
|
||||
if self._move_hide_timer:
|
||||
|
||||
self._move_hide_timer.Start( 250, wx.TIMER_ONE_SHOT )
|
||||
|
||||
|
||||
except:
|
||||
|
||||
if self._move_hide_timer:
|
||||
|
||||
self._move_hide_timer.Stop()
|
||||
|
||||
|
||||
raise
|
||||
|
||||
|
||||
|
||||
def TIMEREventLag( self, event ):
|
||||
|
||||
try:
|
||||
|
||||
self._UpdateList()
|
||||
|
||||
except:
|
||||
|
||||
self._lag_timer.Stop()
|
||||
|
||||
raise
|
||||
|
||||
|
||||
|
||||
class AutoCompleteDropdownTags( AutoCompleteDropdown ):
|
||||
|
||||
def __init__( self, parent, file_service_key, tag_service_key ):
|
||||
|
|
@ -588,7 +587,7 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
|
|||
|
||||
self._file_repo_button.SetLabelText( name )
|
||||
|
||||
wx.CallAfter( self.RefreshList )
|
||||
self._SetListDirty()
|
||||
|
||||
|
||||
def _ChangeTagService( self, tag_service_key ):
|
||||
|
|
@ -610,11 +609,13 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
|
|||
|
||||
self._cache_text = None
|
||||
|
||||
wx.CallAfter( self.RefreshList )
|
||||
self._SetListDirty()
|
||||
|
||||
|
||||
def _UpdateList( self ):
|
||||
|
||||
self._refresh_list_job = None
|
||||
|
||||
self._last_search_text = self._text_ctrl.GetValue()
|
||||
|
||||
matches = self._GenerateMatches()
|
||||
|
|
@ -629,7 +630,9 @@ class AutoCompleteDropdownTags( AutoCompleteDropdown ):
|
|||
|
||||
if num_chars == 0:
|
||||
|
||||
self._lag_timer.Start( 5 * 60 * 1000, wx.TIMER_ONE_SHOT )
|
||||
# refresh system preds after five mins
|
||||
|
||||
self._ScheduleListRefresh( 300 )
|
||||
|
||||
|
||||
|
||||
|
|
@ -1060,7 +1063,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
|
|||
|
||||
self._file_search_context.SetIncludeCurrentTags( value )
|
||||
|
||||
wx.CallAfter( self.RefreshList )
|
||||
self._SetListDirty()
|
||||
|
||||
HG.client_controller.pub( 'refresh_query', self._page_key )
|
||||
|
||||
|
|
@ -1072,7 +1075,7 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
|
|||
|
||||
self._file_search_context.SetIncludePendingTags( value )
|
||||
|
||||
wx.CallAfter( self.RefreshList )
|
||||
self._SetListDirty()
|
||||
|
||||
HG.client_controller.pub( 'refresh_query', self._page_key )
|
||||
|
||||
|
|
|
|||
|
|
@ -1109,7 +1109,7 @@ class CanvasFrame( ClientGUITopLevelWindows.FrameThatResizes ):
|
|||
|
||||
self.SetSizer( vbox )
|
||||
|
||||
ClientGUITopLevelWindows.SetTLWSizeAndPosition( self, self._frame_key )
|
||||
ClientGUITopLevelWindows.SetInitialTLWSizeAndPosition( self, self._frame_key )
|
||||
|
||||
self.Show( True )
|
||||
|
||||
|
|
@ -2757,7 +2757,7 @@ class CanvasWithHovers( CanvasWithDetails ):
|
|||
|
||||
#
|
||||
|
||||
self._timer_cursor_hide = ClientThreading.WXAwareTimer( self, self.TIMERCursorHide )
|
||||
self._timer_cursor_hide_job = None
|
||||
|
||||
self.Bind( wx.EVT_MOTION, self.EventDrag )
|
||||
|
||||
|
|
@ -2863,7 +2863,7 @@ class CanvasWithHovers( CanvasWithDetails ):
|
|||
|
||||
self.SetCursor( wx.Cursor( wx.CURSOR_ARROW ) )
|
||||
|
||||
self._timer_cursor_hide.CallLater( 0.8 )
|
||||
self._PutOffCursorHide()
|
||||
|
||||
else:
|
||||
|
||||
|
|
@ -2871,29 +2871,30 @@ class CanvasWithHovers( CanvasWithDetails ):
|
|||
|
||||
|
||||
|
||||
def TIMERCursorHide( self ):
|
||||
def _PutOffCursorHide( self ):
|
||||
|
||||
try:
|
||||
if self._timer_cursor_hide_job is not None:
|
||||
|
||||
if not CC.CAN_HIDE_MOUSE:
|
||||
|
||||
return
|
||||
|
||||
self._timer_cursor_hide_job.Cancel()
|
||||
|
||||
if HG.client_controller.MenuIsOpen():
|
||||
|
||||
self._timer_cursor_hide.CallLater( 0.8 )
|
||||
|
||||
else:
|
||||
|
||||
self.SetCursor( wx.Cursor( wx.CURSOR_BLANK ) )
|
||||
|
||||
|
||||
self._timer_cursor_hide_job = HG.client_controller.CallLaterWXSafe( self, 0.8, self._HideCursor )
|
||||
|
||||
|
||||
def _HideCursor( self ):
|
||||
|
||||
if not CC.CAN_HIDE_MOUSE:
|
||||
|
||||
except:
|
||||
return
|
||||
|
||||
self._timer_cursor_hide.Stop()
|
||||
|
||||
if HG.client_controller.MenuIsOpen():
|
||||
|
||||
raise
|
||||
self._PutOffCursorHide()
|
||||
|
||||
else:
|
||||
|
||||
self.SetCursor( wx.Cursor( wx.CURSOR_BLANK ) )
|
||||
|
||||
|
||||
|
||||
|
|
@ -3624,20 +3625,17 @@ class CanvasFilterDuplicates( CanvasWithHovers ):
|
|||
|
||||
# ugly, but it will do for now
|
||||
|
||||
if self:
|
||||
if len( self._media_list ) < 2:
|
||||
|
||||
if len( self._media_list ) < 2:
|
||||
|
||||
self._ShowNewPair()
|
||||
|
||||
else:
|
||||
|
||||
self._SetDirty()
|
||||
|
||||
self._ShowNewPair()
|
||||
|
||||
else:
|
||||
|
||||
self._SetDirty()
|
||||
|
||||
|
||||
|
||||
ClientThreading.CallLater( self, 0.1, catch_up )
|
||||
HG.client_controller.CallLaterWXSafe( self, 0.1, catch_up )
|
||||
|
||||
|
||||
def SetMedia( self, media ):
|
||||
|
|
@ -3829,7 +3827,7 @@ class CanvasMediaList( ClientMedia.ListeningMediaList, CanvasWithHovers ):
|
|||
|
||||
if not image_cache.HasImageRenderer( hash ):
|
||||
|
||||
ClientThreading.CallLater( self, delay, image_cache.GetImageRenderer, media )
|
||||
HG.client_controller.CallLaterWXSafe( self, delay, image_cache.GetImageRenderer, media )
|
||||
|
||||
|
||||
|
||||
|
|
@ -3925,7 +3923,7 @@ class CanvasMediaList( ClientMedia.ListeningMediaList, CanvasWithHovers ):
|
|||
|
||||
def KeepCursorAlive( self ):
|
||||
|
||||
self._timer_cursor_hide.CallLater( 0.8 )
|
||||
self._PutOffCursorHide()
|
||||
|
||||
|
||||
def ProcessContentUpdates( self, service_keys_to_content_updates ):
|
||||
|
|
@ -4451,7 +4449,7 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ):
|
|||
|
||||
CanvasMediaListNavigable.__init__( self, parent, page_key, media_results )
|
||||
|
||||
self._timer_slideshow = ClientThreading.WXAwareTimer( self, self.TIMERSlideshow )
|
||||
self._timer_slideshow_job = None
|
||||
self._timer_slideshow_interval = 0
|
||||
|
||||
self.Bind( wx.EVT_LEFT_DCLICK, self.EventClose )
|
||||
|
|
@ -4482,19 +4480,19 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ):
|
|||
|
||||
def _PausePlaySlideshow( self ):
|
||||
|
||||
if self._timer_slideshow.IsRunning():
|
||||
if self._timer_slideshow_job is not None:
|
||||
|
||||
self._timer_slideshow.Stop()
|
||||
self._StopSlideshow()
|
||||
|
||||
elif self._timer_slideshow_interval > 0:
|
||||
|
||||
self._timer_slideshow.CallLater( self._timer_slideshow_interval, repeating = True )
|
||||
self._StartSlideshow( self._timer_slideshow_interval )
|
||||
|
||||
|
||||
|
||||
def _StartSlideshow( self, interval = None ):
|
||||
|
||||
self._timer_slideshow.Stop()
|
||||
self._StopSlideshow()
|
||||
|
||||
if interval is None:
|
||||
|
||||
|
|
@ -4518,7 +4516,43 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ):
|
|||
|
||||
self._timer_slideshow_interval = interval
|
||||
|
||||
self._timer_slideshow.CallLater( self._timer_slideshow_interval, repeating = True )
|
||||
self._timer_slideshow_job = HG.client_controller.CallLaterWXSafe( self, self._timer_slideshow_interval, self.DoSlideshow )
|
||||
|
||||
|
||||
|
||||
def _StopSlideshow( self ):
|
||||
|
||||
if self._timer_slideshow_job is not None:
|
||||
|
||||
self._timer_slideshow_job.Cancel()
|
||||
|
||||
self._timer_slideshow_job = None
|
||||
|
||||
|
||||
|
||||
def DoSlideshow( self ):
|
||||
|
||||
try:
|
||||
|
||||
if self._current_media is not None and self._timer_slideshow_job is not None:
|
||||
|
||||
if self._media_container.ReadyToSlideshow() and not HG.client_controller.MenuIsOpen():
|
||||
|
||||
self._ShowNext()
|
||||
|
||||
self._timer_slideshow_job = HG.client_controller.CallLaterWXSafe( self, self._timer_slideshow_interval, self.DoSlideshow )
|
||||
|
||||
else:
|
||||
|
||||
self._timer_slideshow_job = HG.client_controller.CallLaterWXSafe( self, 0.5, self.DoSlideshow )
|
||||
|
||||
|
||||
|
||||
except:
|
||||
|
||||
self._timer_slideshow_job = None
|
||||
|
||||
raise
|
||||
|
||||
|
||||
|
||||
|
|
@ -4720,7 +4754,7 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ):
|
|||
|
||||
ClientGUIMenus.AppendMenu( menu, slideshow, 'start slideshow' )
|
||||
|
||||
if self._timer_slideshow.IsRunning():
|
||||
if self._timer_slideshow_job is not None:
|
||||
|
||||
ClientGUIMenus.AppendMenuItem( self, menu, 'stop slideshow', 'Stop the current slideshow.', self._PausePlaySlideshow )
|
||||
|
||||
|
|
@ -4742,30 +4776,6 @@ class CanvasMediaListBrowser( CanvasMediaListNavigable ):
|
|||
event.Skip()
|
||||
|
||||
|
||||
def TIMERSlideshow( self ):
|
||||
|
||||
try:
|
||||
|
||||
if self._current_media is not None:
|
||||
|
||||
if self._media_container.ReadyToSlideshow() and not HG.client_controller.MenuIsOpen():
|
||||
|
||||
self._ShowNext()
|
||||
|
||||
else:
|
||||
|
||||
self._timer_slideshow.Delay( 0.5 )
|
||||
|
||||
|
||||
|
||||
except:
|
||||
|
||||
self._timer_slideshow.Stop()
|
||||
|
||||
raise
|
||||
|
||||
|
||||
|
||||
class MediaContainer( wx.Window ):
|
||||
|
||||
def __init__( self, parent ):
|
||||
|
|
|
|||
|
|
@ -664,7 +664,7 @@ class DialogInputLocalBooruShare( Dialog ):
|
|||
|
||||
else:
|
||||
|
||||
time_left = max( 0, timeout - HydrusData.GetNow() )
|
||||
time_left = HydrusData.GetTimeDeltaUntilTime( timeout )
|
||||
|
||||
if time_left < 60 * 60 * 12: time_value = 60
|
||||
elif time_left < 60 * 60 * 24 * 7: time_value = 60 * 60
|
||||
|
|
|
|||
|
|
@ -2411,7 +2411,7 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
|
|||
|
||||
self._import_folder = import_folder
|
||||
|
||||
( name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, open_popup, paused, check_now ) = self._import_folder.ToTuple()
|
||||
( name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, check_regularly, paused, check_now, show_working_popup, publish_files_to_popup_button, publish_files_to_page ) = self._import_folder.ToTuple()
|
||||
|
||||
self._panel = wx.ScrolledWindow( self )
|
||||
|
||||
|
|
@ -2421,7 +2421,7 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
|
|||
|
||||
self._path = wx.DirPickerCtrl( self._folder_box, style = wx.DIRP_USE_TEXTCTRL )
|
||||
|
||||
self._open_popup = wx.CheckBox( self._folder_box )
|
||||
self._check_regularly = wx.CheckBox( self._folder_box )
|
||||
|
||||
self._period = ClientGUITime.TimeDeltaButton( self._folder_box, min = 3 * 60, days = True, hours = True, minutes = True )
|
||||
|
||||
|
|
@ -2429,6 +2429,10 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
|
|||
|
||||
self._check_now = wx.CheckBox( self._folder_box )
|
||||
|
||||
self._show_working_popup = wx.CheckBox( self._folder_box )
|
||||
self._publish_files_to_popup_button = wx.CheckBox( self._folder_box )
|
||||
self._publish_files_to_page = wx.CheckBox( self._folder_box )
|
||||
|
||||
self._seed_cache_button = ClientGUISeedCache.SeedCacheButton( self._folder_box, HG.client_controller, self._import_folder.GetSeedCache, seed_cache_set_callable = self._import_folder.SetSeedCache )
|
||||
|
||||
#
|
||||
|
|
@ -2496,11 +2500,16 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
|
|||
|
||||
self._name.SetValue( name )
|
||||
self._path.SetPath( path )
|
||||
self._open_popup.SetValue( open_popup )
|
||||
|
||||
self._check_regularly.SetValue( check_regularly )
|
||||
|
||||
self._period.SetValue( period )
|
||||
self._paused.SetValue( paused )
|
||||
|
||||
self._show_working_popup.SetValue( show_working_popup )
|
||||
self._publish_files_to_popup_button.SetValue( publish_files_to_popup_button )
|
||||
self._publish_files_to_page.SetValue( publish_files_to_page )
|
||||
|
||||
self._mimes.SetValue( mimes )
|
||||
|
||||
self._action_successful.SelectClientData( actions[ CC.STATUS_SUCCESSFUL ] )
|
||||
|
|
@ -2539,10 +2548,13 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
|
|||
|
||||
rows.append( ( 'name: ', self._name ) )
|
||||
rows.append( ( 'folder path: ', self._path ) )
|
||||
rows.append( ( 'currently paused (if set, will not ever do any work): ', self._paused ) )
|
||||
rows.append( ( 'check regularly?: ', self._check_regularly ) )
|
||||
rows.append( ( 'check period: ', self._period ) )
|
||||
rows.append( ( 'periodic checks currently paused: ', self._paused ) )
|
||||
rows.append( ( 'check on manage dialog ok: ', self._check_now ) )
|
||||
rows.append( ( 'open a popup if new files imported: ', self._open_popup ) )
|
||||
rows.append( ( 'show a popup while working: ', self._show_working_popup ) )
|
||||
rows.append( ( 'if new files imported, publish them to a popup button: ', self._publish_files_to_popup_button ) )
|
||||
rows.append( ( 'if new files imported, publish them to a page: ', self._publish_files_to_page ) )
|
||||
rows.append( ( 'review currently cached import paths: ', self._seed_cache_button ) )
|
||||
|
||||
gridbox = ClientGUICommon.WrapInGrid( self._folder_box, rows )
|
||||
|
|
@ -2621,6 +2633,10 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
|
|||
|
||||
self._CheckLocations()
|
||||
|
||||
self._check_regularly.Bind( wx.EVT_CHECKBOX, self.EventCheckRegularly )
|
||||
|
||||
self._UpdateCheckRegularly()
|
||||
|
||||
wx.CallAfter( self._ok.SetFocus )
|
||||
|
||||
|
||||
|
|
@ -2753,6 +2769,23 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
|
|||
|
||||
|
||||
|
||||
def _UpdateCheckRegularly( self ):
|
||||
|
||||
if self._check_regularly.GetValue():
|
||||
|
||||
self._period.Enable()
|
||||
|
||||
else:
|
||||
|
||||
self._period.Disable()
|
||||
|
||||
|
||||
|
||||
def EventCheckRegularly( self, event ):
|
||||
|
||||
self._UpdateCheckRegularly()
|
||||
|
||||
|
||||
def EventCheckLocations( self, event ):
|
||||
|
||||
self._CheckLocations()
|
||||
|
|
@ -2888,15 +2921,19 @@ class DialogManageImportFoldersEdit( ClientGUIDialogs.Dialog ):
|
|||
|
||||
|
||||
period = self._period.GetValue()
|
||||
open_popup = self._open_popup.GetValue()
|
||||
check_regularly = self._check_regularly.GetValue()
|
||||
|
||||
paused = self._paused.GetValue()
|
||||
|
||||
check_now = self._check_now.GetValue()
|
||||
|
||||
show_working_popup = self._show_working_popup.GetValue()
|
||||
publish_files_to_popup_button = self._publish_files_to_popup_button.GetValue()
|
||||
publish_files_to_page = self._publish_files_to_page.GetValue()
|
||||
|
||||
tag_service_keys_to_filename_tagging_options = dict( self._filename_tagging_options.GetData() )
|
||||
|
||||
self._import_folder.SetTuple( name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, open_popup, paused, check_now )
|
||||
self._import_folder.SetTuple( name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, check_regularly, paused, check_now, show_working_popup, publish_files_to_popup_button, publish_files_to_page )
|
||||
|
||||
return self._import_folder
|
||||
|
||||
|
|
@ -3006,7 +3043,7 @@ class DialogManagePixivAccount( ClientGUIDialogs.Dialog ):
|
|||
|
||||
self._status.SetLabelText( 'OK!' )
|
||||
|
||||
ClientThreading.CallLater( self, 5, self._status.SetLabel, '' )
|
||||
HG.client_controller.CallLaterWXSafe( self._status, 5, self._status.SetLabel, '' )
|
||||
|
||||
else:
|
||||
|
||||
|
|
|
|||
|
|
@ -1207,7 +1207,7 @@ class ManagementPanelDuplicateFilter( ManagementPanel ):
|
|||
time.sleep( 0.25 )
|
||||
|
||||
|
||||
self._RefreshAndUpdateStatus()
|
||||
wx.CallAfter( self._RefreshAndUpdateStatus )
|
||||
|
||||
|
||||
def EventSearchDistanceChanged( self, event ):
|
||||
|
|
@ -2597,36 +2597,6 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
self._controller.sub( self, 'RefreshQuery', 'refresh_query' )
|
||||
|
||||
|
||||
def _BreakApprovedContentsIntoChunks( self, approved_contents ):
|
||||
|
||||
chunks_of_approved_contents = []
|
||||
chunk_of_approved_contents = []
|
||||
weight = 0
|
||||
|
||||
for content in approved_contents:
|
||||
|
||||
chunk_of_approved_contents.append( content )
|
||||
|
||||
weight += content.GetVirtualWeight()
|
||||
|
||||
if weight > 50:
|
||||
|
||||
chunks_of_approved_contents.append( chunk_of_approved_contents )
|
||||
|
||||
chunk_of_approved_contents = []
|
||||
|
||||
weight = 0
|
||||
|
||||
|
||||
|
||||
if len( chunk_of_approved_contents ) > 0:
|
||||
|
||||
chunks_of_approved_contents.append( chunk_of_approved_contents )
|
||||
|
||||
|
||||
return chunks_of_approved_contents
|
||||
|
||||
|
||||
def _CheckAll( self ):
|
||||
|
||||
for i in range( self._contents.GetCount() ):
|
||||
|
|
@ -2754,25 +2724,47 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
|
||||
def _FetchNumPetitions( self ):
|
||||
|
||||
def do_it():
|
||||
def do_it( service ):
|
||||
|
||||
def wx_draw( n_p_i ):
|
||||
|
||||
if not self:
|
||||
|
||||
return
|
||||
|
||||
|
||||
self._num_petition_info = n_p_i
|
||||
|
||||
self._DrawNumPetitions()
|
||||
|
||||
|
||||
def wx_reset():
|
||||
|
||||
if not self:
|
||||
|
||||
return
|
||||
|
||||
|
||||
self._refresh_num_petitions_button.SetLabelText( 'refresh counts' )
|
||||
|
||||
|
||||
try:
|
||||
|
||||
response = self._service.Request( HC.GET, 'num_petitions' )
|
||||
response = service.Request( HC.GET, 'num_petitions' )
|
||||
|
||||
self._num_petition_info = response[ 'num_petitions' ]
|
||||
num_petition_info = response[ 'num_petitions' ]
|
||||
|
||||
wx.CallAfter( self._DrawNumPetitions )
|
||||
wx.CallAfter( wx_draw, num_petition_info )
|
||||
|
||||
finally:
|
||||
|
||||
self._refresh_num_petitions_button.SetLabelText( 'refresh counts' )
|
||||
wx.CallAfter( wx_reset )
|
||||
|
||||
|
||||
|
||||
self._refresh_num_petitions_button.SetLabelText( u'Fetching\u2026' )
|
||||
|
||||
self._controller.CallToThread( do_it )
|
||||
self._controller.CallToThread( do_it, self._service )
|
||||
|
||||
|
||||
def _FetchPetition( self, content_type, status ):
|
||||
|
|
@ -2802,11 +2794,11 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
button.SetLabelText( 'fetch ' + HC.content_status_string_lookup[ status ] + ' ' + HC.content_type_string_lookup[ content_type ] + ' petition' )
|
||||
|
||||
|
||||
def do_it():
|
||||
def do_it( service ):
|
||||
|
||||
try:
|
||||
|
||||
response = self._service.Request( HC.GET, 'petition', { 'content_type' : content_type, 'status' : status } )
|
||||
response = service.Request( HC.GET, 'petition', { 'content_type' : content_type, 'status' : status } )
|
||||
|
||||
wx.CallAfter( wx_setpet, response[ 'petition' ] )
|
||||
|
||||
|
|
@ -2826,7 +2818,7 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
button.Disable()
|
||||
button.SetLabelText( u'Fetching\u2026' )
|
||||
|
||||
self._controller.CallToThread( do_it )
|
||||
self._controller.CallToThread( do_it, self._service )
|
||||
|
||||
|
||||
def _FlipSelected( self ):
|
||||
|
|
@ -2876,7 +2868,37 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
|
||||
def EventProcess( self, event ):
|
||||
|
||||
def do_it( approved_contents, denied_contents, petition ):
|
||||
def break_approved_contents_into_chunks( approved_contents ):
|
||||
|
||||
chunks_of_approved_contents = []
|
||||
chunk_of_approved_contents = []
|
||||
weight = 0
|
||||
|
||||
for content in approved_contents:
|
||||
|
||||
chunk_of_approved_contents.append( content )
|
||||
|
||||
weight += content.GetVirtualWeight()
|
||||
|
||||
if weight > 50:
|
||||
|
||||
chunks_of_approved_contents.append( chunk_of_approved_contents )
|
||||
|
||||
chunk_of_approved_contents = []
|
||||
|
||||
weight = 0
|
||||
|
||||
|
||||
|
||||
if len( chunk_of_approved_contents ) > 0:
|
||||
|
||||
chunks_of_approved_contents.append( chunk_of_approved_contents )
|
||||
|
||||
|
||||
return chunks_of_approved_contents
|
||||
|
||||
|
||||
def do_it( controller, service, petition_service_key, approved_contents, denied_contents, petition ):
|
||||
|
||||
try:
|
||||
|
||||
|
|
@ -2901,7 +2923,7 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
job_key = None
|
||||
|
||||
|
||||
chunks_of_approved_contents = self._BreakApprovedContentsIntoChunks( approved_contents )
|
||||
chunks_of_approved_contents = break_approved_contents_into_chunks( approved_contents )
|
||||
|
||||
for chunk_of_approved_contents in chunks_of_approved_contents:
|
||||
|
||||
|
|
@ -2919,9 +2941,9 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
|
||||
( update, content_updates ) = petition.GetApproval( chunk_of_approved_contents )
|
||||
|
||||
self._service.Request( HC.POST, 'update', { 'client_to_server_update' : update } )
|
||||
service.Request( HC.POST, 'update', { 'client_to_server_update' : update } )
|
||||
|
||||
self._controller.WriteSynchronous( 'content_updates', { self._petition_service_key : content_updates } )
|
||||
controller.WriteSynchronous( 'content_updates', { petition_service_key : content_updates } )
|
||||
|
||||
num_done += len( chunk_of_approved_contents )
|
||||
|
||||
|
|
@ -2940,7 +2962,7 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
|
||||
update = petition.GetDenial( denied_contents )
|
||||
|
||||
self._service.Request( HC.POST, 'update', { 'client_to_server_update' : update } )
|
||||
service.Request( HC.POST, 'update', { 'client_to_server_update' : update } )
|
||||
|
||||
|
||||
finally:
|
||||
|
|
@ -2950,7 +2972,17 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
job_key.Delete()
|
||||
|
||||
|
||||
wx.CallAfter( self._FetchNumPetitions )
|
||||
def wx_fetch():
|
||||
|
||||
if not self:
|
||||
|
||||
return
|
||||
|
||||
|
||||
self._FetchNumPetitions()
|
||||
|
||||
|
||||
wx.CallAfter( wx_fetch )
|
||||
|
||||
|
||||
|
||||
|
|
@ -2971,7 +3003,7 @@ class ManagementPanelPetitions( ManagementPanel ):
|
|||
|
||||
|
||||
|
||||
HG.client_controller.CallToThread( do_it, approved_contents, denied_contents, self._current_petition )
|
||||
HG.client_controller.CallToThread( do_it, self._controller, self._service, self._petition_service_key, approved_contents, denied_contents, self._current_petition )
|
||||
|
||||
self._current_petition = None
|
||||
|
||||
|
|
@ -3133,11 +3165,6 @@ class ManagementPanelQuery( ManagementPanel ):
|
|||
|
||||
self._query_job_key.Cancel()
|
||||
|
||||
if self._search_enabled:
|
||||
|
||||
self._searchbox.CleanBeforeDestroy()
|
||||
|
||||
|
||||
|
||||
def GetPredicates( self ):
|
||||
|
||||
|
|
|
|||
|
|
@ -455,7 +455,7 @@ class Page( wx.SplitterWindow ):
|
|||
self._media_panel.Hide()
|
||||
|
||||
# If this is a CallAfter, OS X segfaults on refresh jej
|
||||
ClientThreading.CallLater( self, 0.5, self._media_panel.Destroy )
|
||||
self._controller.CallLaterWXSafe( self._media_panel, 0.5, self._media_panel.Destroy )
|
||||
|
||||
self._media_panel = new_panel
|
||||
|
||||
|
|
@ -1059,6 +1059,29 @@ class PagesNotebook( wx.Notebook ):
|
|||
|
||||
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _GetPageFromName( self, page_name ):
|
||||
|
||||
for page in self._GetPages():
|
||||
|
||||
if page.GetDisplayName() == page_name:
|
||||
|
||||
return page
|
||||
|
||||
|
||||
if isinstance( page, PagesNotebook ):
|
||||
|
||||
result = page._GetPageFromName( page_name )
|
||||
|
||||
if result is not None:
|
||||
|
||||
return result
|
||||
|
||||
|
||||
|
||||
|
||||
return None
|
||||
|
||||
|
||||
|
|
@ -1934,7 +1957,7 @@ class PagesNotebook( wx.Notebook ):
|
|||
|
||||
|
||||
|
||||
def NewPage( self, management_controller, initial_hashes = None, forced_insertion_index = None, on_deepest_notebook = False ):
|
||||
def NewPage( self, management_controller, initial_hashes = None, forced_insertion_index = None, on_deepest_notebook = False, select_page = True ):
|
||||
|
||||
current_page = self.GetCurrentPage()
|
||||
|
||||
|
|
@ -2013,7 +2036,7 @@ class PagesNotebook( wx.Notebook ):
|
|||
# in some unusual circumstances, this gets out of whack
|
||||
insertion_index = min( insertion_index, self.GetPageCount() )
|
||||
|
||||
self.InsertPage( insertion_index, page, page_name, select = True )
|
||||
self.InsertPage( insertion_index, page, page_name, select = select_page )
|
||||
|
||||
self._controller.pub( 'refresh_page_name', page.GetPageKey() )
|
||||
self._controller.pub( 'notify_new_pages' )
|
||||
|
|
@ -2079,7 +2102,7 @@ class PagesNotebook( wx.Notebook ):
|
|||
return self.NewPage( management_controller, on_deepest_notebook = on_deepest_notebook )
|
||||
|
||||
|
||||
def NewPageQuery( self, file_service_key, initial_hashes = None, initial_predicates = None, page_name = None, on_deepest_notebook = False, do_sort = False ):
|
||||
def NewPageQuery( self, file_service_key, initial_hashes = None, initial_predicates = None, page_name = None, on_deepest_notebook = False, do_sort = False, select_page = True ):
|
||||
|
||||
if initial_hashes is None:
|
||||
|
||||
|
|
@ -2111,7 +2134,7 @@ class PagesNotebook( wx.Notebook ):
|
|||
|
||||
management_controller = ClientGUIManagement.CreateManagementControllerQuery( page_name, file_service_key, file_search_context, search_enabled )
|
||||
|
||||
page = self.NewPage( management_controller, initial_hashes = initial_hashes, on_deepest_notebook = on_deepest_notebook )
|
||||
page = self.NewPage( management_controller, initial_hashes = initial_hashes, on_deepest_notebook = on_deepest_notebook, select_page = select_page )
|
||||
|
||||
if do_sort:
|
||||
|
||||
|
|
@ -2338,6 +2361,28 @@ class PagesNotebook( wx.Notebook ):
|
|||
|
||||
|
||||
|
||||
def PresentImportedFilesToPage( self, hashes, page_name ):
|
||||
|
||||
page = self._GetPageFromName( page_name )
|
||||
|
||||
if page is None:
|
||||
|
||||
page = self.NewPageQuery( CC.LOCAL_FILE_SERVICE_KEY, initial_hashes = hashes, page_name = page_name, on_deepest_notebook = True, select_page = False )
|
||||
|
||||
else:
|
||||
|
||||
unsorted_media_results = self._controller.Read( 'media_results', hashes )
|
||||
|
||||
hashes_to_media_results = { media_result.GetHash() : media_result for media_result in unsorted_media_results }
|
||||
|
||||
sorted_media_results = [ hashes_to_media_results[ hash ] for hash in hashes ]
|
||||
|
||||
page.GetMediaPanel().AddMediaResults( page.GetPageKey(), sorted_media_results )
|
||||
|
||||
|
||||
return page
|
||||
|
||||
|
||||
def RefreshAllPages( self ):
|
||||
|
||||
for page in self._GetPages():
|
||||
|
|
|
|||
|
|
@ -218,7 +218,7 @@ class EditCompoundFormulaPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
The substitution phrase works like in regexes. If you have two formulae that produce _filename_ and _ext_, you could set this:
|
||||
|
||||
\1.\2'''
|
||||
\\1.\\2'''
|
||||
|
||||
info_st = wx.StaticText( info_panel, label = message )
|
||||
|
||||
|
|
@ -903,9 +903,9 @@ The html's branches will be searched recursively by each tag rule in turn and th
|
|||
|
||||
So, to find the 'src' of the first <img> tag beneath all <span> tags with the class 'content', use:
|
||||
|
||||
'all span tags with class=content'
|
||||
1st img tag'
|
||||
attribute: src'
|
||||
all span tags with class=content
|
||||
1st img tag
|
||||
attribute: src
|
||||
|
||||
'''
|
||||
|
||||
|
|
|
|||
|
|
@ -560,23 +560,15 @@ class PopupMessageManager( wx.Frame ):
|
|||
HydrusData.ShowException = ClientData.ShowExceptionClient
|
||||
HydrusData.ShowText = ClientData.ShowTextClient
|
||||
|
||||
self.Bind( wx.EVT_TIMER, self.TIMEREvent )
|
||||
|
||||
self._timer = wx.Timer( self )
|
||||
|
||||
self._timer.Start( 500, wx.TIMER_CONTINUOUS )
|
||||
|
||||
job_key = ClientThreading.JobKey()
|
||||
|
||||
job_key.SetVariable( 'popup_text_1', u'initialising popup message manager\u2026' )
|
||||
|
||||
wx.CallAfter( self.AddMessage, job_key )
|
||||
self._update_job = HG.client_controller.CallRepeatingWXSafe( self, 0.5, 0.25, self.REPEATINGUpdate )
|
||||
|
||||
wx.CallAfter( self._Update )
|
||||
HG.client_controller.CallLaterWXSafe( self, 0.5, self.AddMessage, job_key )
|
||||
|
||||
wx.CallAfter( job_key.Delete )
|
||||
|
||||
wx.CallAfter( self._Update )
|
||||
HG.client_controller.CallLaterWXSafe( self, 1.0, job_key.Delete )
|
||||
|
||||
|
||||
def _CheckPending( self ):
|
||||
|
|
@ -772,7 +764,7 @@ class PopupMessageManager( wx.Frame ):
|
|||
|
||||
wx.MessageBox( text )
|
||||
|
||||
self._timer.Stop()
|
||||
self._update_job.Cancel()
|
||||
|
||||
self.CleanBeforeDestroy()
|
||||
|
||||
|
|
@ -780,11 +772,79 @@ class PopupMessageManager( wx.Frame ):
|
|||
|
||||
|
||||
|
||||
def _GetAllMessageJobKeys( self ):
|
||||
|
||||
job_keys = []
|
||||
|
||||
sizer_items = self._message_vbox.GetChildren()
|
||||
|
||||
for sizer_item in sizer_items:
|
||||
|
||||
message_window = sizer_item.GetWindow()
|
||||
|
||||
job_key = message_window.GetJobKey()
|
||||
|
||||
job_keys.append( job_key )
|
||||
|
||||
|
||||
job_keys.extend( self._pending_job_keys )
|
||||
|
||||
return job_keys
|
||||
|
||||
|
||||
def _TryToMergeMessage( self, job_key ):
|
||||
|
||||
if not job_key.HasVariable( 'popup_files_mergable' ):
|
||||
|
||||
return False
|
||||
|
||||
|
||||
result = job_key.GetIfHasVariable( 'popup_files' )
|
||||
|
||||
if result is not None:
|
||||
|
||||
( hashes, name ) = result
|
||||
|
||||
existing_job_keys = self._GetAllMessageJobKeys()
|
||||
|
||||
for existing_job_key in existing_job_keys:
|
||||
|
||||
if existing_job_key.HasVariable( 'popup_files_mergable' ):
|
||||
|
||||
result = existing_job_key.GetIfHasVariable( 'popup_files' )
|
||||
|
||||
if result is not None:
|
||||
|
||||
( existing_hashes, existing_name ) = result
|
||||
|
||||
if existing_name == name:
|
||||
|
||||
if isinstance( existing_hashes, list ):
|
||||
|
||||
existing_hashes.extend( hashes )
|
||||
|
||||
elif isinstance( existing_hashes, set ):
|
||||
|
||||
existing_hashes.update( hashes )
|
||||
|
||||
|
||||
return True
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def _Update( self ):
|
||||
|
||||
if HG.view_shutdown:
|
||||
|
||||
self._timer.Stop()
|
||||
self._update_job.Cancel()
|
||||
|
||||
self.CleanBeforeDestroy()
|
||||
|
||||
self.Destroy()
|
||||
|
||||
|
|
@ -816,6 +876,13 @@ class PopupMessageManager( wx.Frame ):
|
|||
|
||||
try:
|
||||
|
||||
was_merged = self._TryToMergeMessage( job_key )
|
||||
|
||||
if was_merged:
|
||||
|
||||
return
|
||||
|
||||
|
||||
self._pending_job_keys.append( job_key )
|
||||
|
||||
if ClientGUITopLevelWindows.MouseIsOnMyDisplay( self.GetParent() ):
|
||||
|
|
@ -853,13 +920,6 @@ class PopupMessageManager( wx.Frame ):
|
|||
|
||||
|
||||
|
||||
if self._timer is not None:
|
||||
|
||||
self._timer.Stop()
|
||||
|
||||
self._timer = None
|
||||
|
||||
|
||||
sys.excepthook = self._old_excepthook
|
||||
|
||||
HydrusData.ShowException = self._old_show_exception
|
||||
|
|
@ -905,7 +965,7 @@ class PopupMessageManager( wx.Frame ):
|
|||
self._SizeAndPositionAndShow()
|
||||
|
||||
|
||||
def TIMEREvent( self, event ):
|
||||
def REPEATINGUpdate( self ):
|
||||
|
||||
try:
|
||||
|
||||
|
|
@ -918,7 +978,7 @@ class PopupMessageManager( wx.Frame ):
|
|||
|
||||
except:
|
||||
|
||||
self._timer.Stop()
|
||||
self._update_job.Cancel()
|
||||
|
||||
raise
|
||||
|
||||
|
|
@ -940,27 +1000,13 @@ class PopupMessageDialogPanel( ClientGUIScrolledPanels.ReviewPanelVetoable ):
|
|||
|
||||
self.SetSizer( vbox )
|
||||
|
||||
self.Bind( wx.EVT_TIMER, self.TIMEREvent )
|
||||
|
||||
self._windows_minimised = []
|
||||
|
||||
self._MinimiseOtherWindows()
|
||||
|
||||
self._timer = wx.Timer( self )
|
||||
|
||||
self._timer.Start( 500, wx.TIMER_CONTINUOUS )
|
||||
|
||||
self._message_pubbed = False
|
||||
|
||||
|
||||
def _DestroyTimer( self ):
|
||||
|
||||
if self._timer is not None:
|
||||
|
||||
self._timer.Stop()
|
||||
|
||||
self._timer = None
|
||||
|
||||
self._update_job = HG.client_controller.CallRepeatingWXSafe( self, 0.5, 0.25, self.REPEATINGUpdate )
|
||||
|
||||
|
||||
def _MinimiseOtherWindows( self ):
|
||||
|
|
@ -1040,8 +1086,6 @@ class PopupMessageDialogPanel( ClientGUIScrolledPanels.ReviewPanelVetoable ):
|
|||
|
||||
self._ReleaseMessage()
|
||||
|
||||
self._DestroyTimer()
|
||||
|
||||
else:
|
||||
|
||||
if self._job_key.IsCancellable():
|
||||
|
|
@ -1054,8 +1098,6 @@ class PopupMessageDialogPanel( ClientGUIScrolledPanels.ReviewPanelVetoable ):
|
|||
|
||||
self._ReleaseMessage()
|
||||
|
||||
self._DestroyTimer()
|
||||
|
||||
else:
|
||||
|
||||
raise HydrusExceptions.VetoException()
|
||||
|
|
@ -1069,7 +1111,7 @@ class PopupMessageDialogPanel( ClientGUIScrolledPanels.ReviewPanelVetoable ):
|
|||
|
||||
|
||||
|
||||
def TIMEREvent( self, event ):
|
||||
def REPEATINGUpdate( self ):
|
||||
|
||||
try:
|
||||
|
||||
|
|
@ -1089,7 +1131,7 @@ class PopupMessageDialogPanel( ClientGUIScrolledPanels.ReviewPanelVetoable ):
|
|||
|
||||
except:
|
||||
|
||||
self._DestroyTimer()
|
||||
self._update_job.Cancel()
|
||||
|
||||
raise
|
||||
|
||||
|
|
|
|||
|
|
@ -29,6 +29,7 @@ import HydrusNetwork
|
|||
import HydrusSerialisable
|
||||
import HydrusText
|
||||
import os
|
||||
import webbrowser
|
||||
import wx
|
||||
|
||||
class EditAccountTypePanel( ClientGUIScrolledPanels.EditPanel ):
|
||||
|
|
@ -3411,6 +3412,14 @@ class EditURLMatchesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
ClientGUIScrolledPanels.EditPanel.__init__( self, parent )
|
||||
|
||||
menu_items = []
|
||||
|
||||
page_func = HydrusData.Call( webbrowser.open, 'file://' + HC.HELP_DIR + '/downloader_url_classes.html' )
|
||||
|
||||
menu_items.append( ( 'normal', 'open the url classes help', 'Open the help page for url classes in your web browesr.', page_func ) )
|
||||
|
||||
help_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.help, menu_items )
|
||||
|
||||
self._list_ctrl_panel = ClientGUIListCtrl.BetterListCtrlPanel( self )
|
||||
|
||||
self._list_ctrl = ClientGUIListCtrl.BetterListCtrl( self._list_ctrl_panel, 'url_matches', 15, 40, [ ( 'name', 36 ), ( 'type', 20 ), ( 'example url', -1 ) ], self._ConvertDataToListCtrlTuples, delete_key_callback = self._Delete, activation_callback = self._Edit )
|
||||
|
|
@ -3433,8 +3442,18 @@ class EditURLMatchesPanel( ClientGUIScrolledPanels.EditPanel ):
|
|||
|
||||
#
|
||||
|
||||
help_hbox = wx.BoxSizer( wx.HORIZONTAL )
|
||||
|
||||
st = ClientGUICommon.BetterStaticText( self, 'help for this panel -->' )
|
||||
|
||||
st.SetForegroundColour( wx.Colour( 0, 0, 255 ) )
|
||||
|
||||
help_hbox.Add( st, CC.FLAGS_VCENTER )
|
||||
help_hbox.Add( help_button, CC.FLAGS_VCENTER )
|
||||
|
||||
vbox = wx.BoxSizer( wx.VERTICAL )
|
||||
|
||||
vbox.Add( help_hbox, CC.FLAGS_BUTTON_SIZER )
|
||||
vbox.Add( self._list_ctrl_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
|
||||
|
||||
self.SetSizer( vbox )
|
||||
|
|
|
|||
|
|
@ -5076,7 +5076,7 @@ class ManageSubscriptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
|
|||
else:
|
||||
|
||||
pretty_delay = 'delaying ' + HydrusData.ConvertTimestampToPrettyPending( no_work_until, prefix = 'for' ) + ' - ' + no_work_until_reason
|
||||
delay = no_work_until - HydrusData.GetNow()
|
||||
delay = HydrusData.GetTimeDeltaUntilTime( no_work_until )
|
||||
|
||||
|
||||
num_urls_done = 0
|
||||
|
|
|
|||
|
|
@ -332,9 +332,7 @@ class ReviewAllBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
self._bandwidths.Sort( 0 )
|
||||
|
||||
self._update_timer = ClientThreading.WXAwareTimer( self, self._Update )
|
||||
|
||||
self._Update()
|
||||
self._update_job = HG.client_controller.CallRepeatingWXSafe( self, 5.0, 0.0, self._Update )
|
||||
|
||||
#
|
||||
|
||||
|
|
@ -422,7 +420,7 @@ class ReviewAllBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
self._controller.network_engine.bandwidth_manager.DeleteHistory( selected_network_contexts )
|
||||
|
||||
self._Update()
|
||||
self._update_job.MoveNextWorkTimeToNow()
|
||||
|
||||
|
||||
|
||||
|
|
@ -479,8 +477,9 @@ class ReviewAllBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
help += os.linesep * 2
|
||||
help += 'There are two special \'instance\' contexts, for downloaders and threads. These represent individual queries, either a single gallery search or a single watched thread. It is useful to set rules for these so your searches will gather a fast initial sample of results in the first few minutes--so you can make sure you are happy with them--but otherwise trickle the rest in over time. This keeps your CPU and other bandwidth limits less hammered and helps to avoid accidental downloads of many thousands of small bad files or a few hundred gigantic files all in one go.'
|
||||
help += os.linesep * 2
|
||||
help += 'If you do not understand what is going on here, you can safely leave it alone. The default settings make for a _reasonable_ and polite profile that will not accidentally cause you to download way too much in one go or piss off servers by being too aggressive. The simplest way of throttling your client is by editing the rules for the global context.'
|
||||
|
||||
help += 'Please note that this system bases its calendar dates on UTC/GMT time (it helps servers and clients around the world stay in sync a bit easier). This has no bearing on what, for instance, the \'past 24 hours\' means, but monthly transitions may occur a few hours off whatever your midnight is.'
|
||||
help += os.linesep * 2
|
||||
help += 'If you do not understand what is going on here, you can safely leave it alone. The default settings make for a _reasonable_ and polite profile that will not accidentally cause you to download way too much in one go or piss off servers by being too aggressive. If you want to throttle your client, the simplest way is to add a simple rule like \'500MB per day\' to the global context.'
|
||||
wx.MessageBox( help )
|
||||
|
||||
|
||||
|
|
@ -501,8 +500,6 @@ class ReviewAllBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
timer_duration_s = max( len( network_contexts ), 20 )
|
||||
|
||||
self._update_timer.CallLater( timer_duration_s )
|
||||
|
||||
|
||||
def EventTimeDeltaChanged( self, event ):
|
||||
|
||||
|
|
@ -515,7 +512,7 @@ class ReviewAllBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
self._history_time_delta_threshold.Enable()
|
||||
|
||||
|
||||
self._Update()
|
||||
self._update_job.MoveNextWorkTimeToNow()
|
||||
|
||||
|
||||
def ShowNetworkContext( self ):
|
||||
|
|
@ -645,12 +642,9 @@ class ReviewNetworkContextBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
#
|
||||
|
||||
self._rules_timer = ClientThreading.WXAwareTimer( self, self._UpdateRules )
|
||||
self._rules_job = HG.client_controller.CallRepeatingWXSafe( self, 5.0, 0.0, self._UpdateRules )
|
||||
|
||||
self._update_timer = ClientThreading.WXAwareTimer( self, self._Update )
|
||||
|
||||
self._UpdateRules()
|
||||
self._Update()
|
||||
self._update_job = HG.client_controller.CallRepeatingWXSafe( self, 1.0, 0.0, self._Update )
|
||||
|
||||
|
||||
def _EditRules( self ):
|
||||
|
|
@ -700,8 +694,6 @@ class ReviewNetworkContextBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
self._time_delta_usage_st.SetLabelText( pretty_time_delta_usage )
|
||||
|
||||
self._update_timer.CallLater( 1.0 )
|
||||
|
||||
|
||||
def _UpdateRules( self ):
|
||||
|
||||
|
|
@ -778,8 +770,6 @@ class ReviewNetworkContextBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
ClientGUITopLevelWindows.PostSizeChangedEvent( self )
|
||||
|
||||
|
||||
self._rules_timer.CallLater( 5.0 )
|
||||
|
||||
|
||||
def _UseDefaultRules( self ):
|
||||
|
||||
|
|
@ -789,7 +779,7 @@ class ReviewNetworkContextBandwidthPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
self._controller.network_engine.bandwidth_manager.DeleteRules( self._network_context )
|
||||
|
||||
self._UpdateRules()
|
||||
self._rules_job.MoveNextWorkTimeToNow()
|
||||
|
||||
|
||||
|
||||
|
|
@ -1718,7 +1708,7 @@ def THREADMigrateDatabase( controller, source, portable_locations, dest ):
|
|||
|
||||
def wx_code( job_key ):
|
||||
|
||||
ClientThreading.CallLater( controller.gui, 3, controller.gui.Exit )
|
||||
HG.client_controller.CallLaterWXSafe( controller.gui, 3.0, controller.gui.Exit )
|
||||
|
||||
# no parent because this has to outlive the gui, obvs
|
||||
|
||||
|
|
|
|||
|
|
@ -591,7 +591,7 @@ class SeedCacheStatusControl( wx.Panel ):
|
|||
|
||||
#
|
||||
|
||||
self._update_timer = ClientThreading.WXAwareTimer( self, self.TIMERUpdate )
|
||||
HG.client_controller.gui.RegisterUIUpdateWindow( self )
|
||||
|
||||
|
||||
def _GetSeedCache( self ):
|
||||
|
|
@ -638,29 +638,17 @@ class SeedCacheStatusControl( wx.Panel ):
|
|||
|
||||
|
||||
|
||||
def ClearSeedCache( self ):
|
||||
|
||||
if self:
|
||||
|
||||
self._Update()
|
||||
|
||||
self._seed_cache = None
|
||||
|
||||
self._update_timer.Stop()
|
||||
|
||||
|
||||
|
||||
def SetSeedCache( self, seed_cache ):
|
||||
|
||||
if self:
|
||||
if not self:
|
||||
|
||||
self._seed_cache = seed_cache
|
||||
|
||||
self._update_timer.CallLater( 0.25, repeating = True )
|
||||
return
|
||||
|
||||
|
||||
self._seed_cache = seed_cache
|
||||
|
||||
|
||||
def TIMERUpdate( self ):
|
||||
def TIMERUIUpdate( self ):
|
||||
|
||||
if self._controller.gui.IShouldRegularlyUpdate( self ):
|
||||
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import ClientSerialisable
|
|||
import ClientThreading
|
||||
import HydrusConstants as HC
|
||||
import HydrusData
|
||||
import HydrusGlobals as HG
|
||||
import HydrusSerialisable
|
||||
import os
|
||||
import wx
|
||||
|
|
@ -119,7 +120,7 @@ class PngExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
self._export.SetLabelText( 'done!' )
|
||||
|
||||
ClientThreading.CallLater( self, 2, self._export.SetLabelText, 'export' )
|
||||
HG.client_controller.CallLaterWXSafe( self._export, 2.0, self._export.SetLabelText, 'export' )
|
||||
|
||||
|
||||
class PngsExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
||||
|
|
@ -210,6 +211,6 @@ class PngsExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
|
|||
|
||||
self._export.SetLabelText( 'done!' )
|
||||
|
||||
ClientThreading.CallLater( self, 2, self._export.SetLabelText, 'export' )
|
||||
HG.client_controller.CallLaterWXSafe( self._export, 2.0, self._export.SetLabelText, 'export' )
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -171,7 +171,7 @@ def SaveTLWSizeAndPosition( tlw, frame_key ):
|
|||
|
||||
new_options.SetFrameLocation( frame_key, remember_size, remember_position, last_size, last_position, default_gravity, default_position, maximised, fullscreen )
|
||||
|
||||
def SetTLWSizeAndPosition( tlw, frame_key ):
|
||||
def SetInitialTLWSizeAndPosition( tlw, frame_key ):
|
||||
|
||||
new_options = HG.client_controller.new_options
|
||||
|
||||
|
|
@ -482,7 +482,7 @@ class DialogThatTakesScrollablePanel( DialogThatResizes ):
|
|||
|
||||
self.SetSizer( vbox )
|
||||
|
||||
SetTLWSizeAndPosition( self, self._frame_key )
|
||||
SetInitialTLWSizeAndPosition( self, self._frame_key )
|
||||
|
||||
self._panel.SetupScrolling() # this changes geteffectiveminsize calc, so it needs to be below settlwsizeandpos
|
||||
|
||||
|
|
@ -834,10 +834,12 @@ class FrameThatTakesScrollablePanel( FrameThatResizes ):
|
|||
|
||||
self.SetSizer( vbox )
|
||||
|
||||
SetTLWSizeAndPosition( self, self._frame_key )
|
||||
SetInitialTLWSizeAndPosition( self, self._frame_key )
|
||||
|
||||
self.Show( True )
|
||||
|
||||
self._panel.SetupScrolling()
|
||||
|
||||
PostSizeChangedEvent( self ) # helps deal with some Linux/otherscrollbar weirdness where setupscrolling changes inherant virtual size
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -2001,9 +2001,9 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_IMPORT_FOLDER
|
||||
SERIALISABLE_NAME = 'Import Folder'
|
||||
SERIALISABLE_VERSION = 5
|
||||
SERIALISABLE_VERSION = 6
|
||||
|
||||
def __init__( self, name, path = '', file_import_options = None, tag_import_options = None, tag_service_keys_to_filename_tagging_options = None, mimes = None, actions = None, action_locations = None, period = 3600, open_popup = True ):
|
||||
def __init__( self, name, path = '', file_import_options = None, tag_import_options = None, tag_service_keys_to_filename_tagging_options = None, mimes = None, actions = None, action_locations = None, period = 3600, check_regularly = True, show_working_popup = True, publish_files_to_popup_button = True, publish_files_to_page = False ):
|
||||
|
||||
if mimes is None:
|
||||
|
||||
|
|
@ -2050,13 +2050,17 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
self._actions = actions
|
||||
self._action_locations = action_locations
|
||||
self._period = period
|
||||
self._open_popup = open_popup
|
||||
self._check_regularly = check_regularly
|
||||
|
||||
self._path_cache = SeedCache()
|
||||
self._last_checked = 0
|
||||
self._paused = False
|
||||
self._check_now = False
|
||||
|
||||
self._show_working_popup = show_working_popup
|
||||
self._publish_files_to_popup_button = publish_files_to_popup_button
|
||||
self._publish_files_to_page = publish_files_to_page
|
||||
|
||||
|
||||
def _ActionPaths( self ):
|
||||
|
||||
|
|
@ -2176,6 +2180,44 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
|
||||
|
||||
def _CheckFolder( self, job_key ):
|
||||
|
||||
filenames = os.listdir( HydrusData.ToUnicode( self._path ) )
|
||||
|
||||
raw_paths = [ os.path.join( self._path, filename ) for filename in filenames ]
|
||||
|
||||
all_paths = ClientFiles.GetAllPaths( raw_paths )
|
||||
|
||||
all_paths = HydrusPaths.FilterFreePaths( all_paths )
|
||||
|
||||
new_paths = []
|
||||
|
||||
for path in all_paths:
|
||||
|
||||
if job_key.IsCancelled():
|
||||
|
||||
break
|
||||
|
||||
|
||||
if path.endswith( '.txt' ):
|
||||
|
||||
continue
|
||||
|
||||
|
||||
if not self._path_cache.HasPath( path ):
|
||||
|
||||
new_paths.append( path )
|
||||
|
||||
|
||||
job_key.SetVariable( 'popup_text_1', 'checking: found ' + HydrusData.ConvertIntToPrettyString( len( new_paths ) ) + ' new files' )
|
||||
|
||||
|
||||
self._path_cache.AddPaths( new_paths )
|
||||
|
||||
self._last_checked = HydrusData.GetNow()
|
||||
self._check_now = False
|
||||
|
||||
|
||||
def _GetSerialisableInfo( self ):
|
||||
|
||||
serialisable_file_import_options = self._file_import_options.GetSerialisableTuple()
|
||||
|
|
@ -2187,12 +2229,198 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
action_pairs = self._actions.items()
|
||||
action_location_pairs = self._action_locations.items()
|
||||
|
||||
return ( self._path, self._mimes, serialisable_file_import_options, serialisable_tag_import_options, serialisable_tag_service_keys_to_filename_tagging_options, action_pairs, action_location_pairs, self._period, self._open_popup, serialisable_path_cache, self._last_checked, self._paused, self._check_now )
|
||||
return ( self._path, self._mimes, serialisable_file_import_options, serialisable_tag_import_options, serialisable_tag_service_keys_to_filename_tagging_options, action_pairs, action_location_pairs, self._period, self._check_regularly, serialisable_path_cache, self._last_checked, self._paused, self._check_now, self._show_working_popup, self._publish_files_to_popup_button, self._publish_files_to_page )
|
||||
|
||||
|
||||
def _ImportFiles( self, job_key ):
|
||||
|
||||
did_work = False
|
||||
|
||||
time_to_save = HydrusData.GetNow() + 600
|
||||
|
||||
num_files_imported = 0
|
||||
presentation_hashes = []
|
||||
presentation_hashes_fast = set()
|
||||
|
||||
i = 0
|
||||
|
||||
num_total = len( self._path_cache )
|
||||
num_total_unknown = self._path_cache.GetSeedCount( CC.STATUS_UNKNOWN )
|
||||
num_total_done = num_total - num_total_unknown
|
||||
|
||||
while True:
|
||||
|
||||
seed = self._path_cache.GetNextSeed( CC.STATUS_UNKNOWN )
|
||||
|
||||
p1 = HC.options[ 'pause_import_folders_sync' ] or self._paused
|
||||
p2 = HydrusThreading.IsThreadShuttingDown()
|
||||
p3 = job_key.IsCancelled()
|
||||
|
||||
if seed is None or p1 or p2 or p3:
|
||||
|
||||
break
|
||||
|
||||
|
||||
if HydrusData.TimeHasPassed( time_to_save ):
|
||||
|
||||
HG.client_controller.WriteSynchronous( 'serialisable', self )
|
||||
|
||||
time_to_save = HydrusData.GetNow() + 600
|
||||
|
||||
|
||||
gauge_num_done = num_total_done + num_files_imported + 1
|
||||
|
||||
job_key.SetVariable( 'popup_text_1', 'importing file ' + HydrusData.ConvertValueRangeToPrettyString( gauge_num_done, num_total ) )
|
||||
job_key.SetVariable( 'popup_gauge_1', ( gauge_num_done, num_total ) )
|
||||
|
||||
path = seed.seed_data
|
||||
|
||||
try:
|
||||
|
||||
mime = HydrusFileHandling.GetMime( path )
|
||||
|
||||
if mime in self._mimes:
|
||||
|
||||
( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
|
||||
|
||||
try:
|
||||
|
||||
copied = HydrusPaths.MirrorFile( path, temp_path )
|
||||
|
||||
if not copied:
|
||||
|
||||
raise Exception( 'File failed to copy--see log for error.' )
|
||||
|
||||
|
||||
file_import_job = FileImportJob( temp_path, self._file_import_options )
|
||||
|
||||
client_files_manager = HG.client_controller.client_files_manager
|
||||
|
||||
( status, hash ) = client_files_manager.ImportFile( file_import_job )
|
||||
|
||||
finally:
|
||||
|
||||
HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
|
||||
|
||||
|
||||
seed.SetStatus( status )
|
||||
|
||||
if status in ( CC.STATUS_SUCCESSFUL, CC.STATUS_REDUNDANT ):
|
||||
|
||||
downloaded_tags = []
|
||||
|
||||
service_keys_to_content_updates = self._tag_import_options.GetServiceKeysToContentUpdates( hash, downloaded_tags ) # explicit tags
|
||||
|
||||
if len( service_keys_to_content_updates ) > 0:
|
||||
|
||||
HG.client_controller.WriteSynchronous( 'content_updates', service_keys_to_content_updates )
|
||||
|
||||
|
||||
service_keys_to_tags = {}
|
||||
|
||||
for ( tag_service_key, filename_tagging_options ) in self._tag_service_keys_to_filename_tagging_options.items():
|
||||
|
||||
if not HG.client_controller.services_manager.ServiceExists( tag_service_key ):
|
||||
|
||||
continue
|
||||
|
||||
|
||||
try:
|
||||
|
||||
tags = filename_tagging_options.GetTags( tag_service_key, path )
|
||||
|
||||
if len( tags ) > 0:
|
||||
|
||||
service_keys_to_tags[ tag_service_key ] = tags
|
||||
|
||||
|
||||
except Exception as e:
|
||||
|
||||
HydrusData.ShowText( 'Trying to parse filename tags in the import folder "' + self._name + '" threw an error!' )
|
||||
|
||||
HydrusData.ShowException( e )
|
||||
|
||||
|
||||
|
||||
if len( service_keys_to_tags ) > 0:
|
||||
|
||||
service_keys_to_content_updates = ClientData.ConvertServiceKeysToTagsToServiceKeysToContentUpdates( { hash }, service_keys_to_tags )
|
||||
|
||||
HG.client_controller.WriteSynchronous( 'content_updates', service_keys_to_content_updates )
|
||||
|
||||
|
||||
num_files_imported += 1
|
||||
|
||||
if hash not in presentation_hashes_fast:
|
||||
|
||||
in_inbox = HG.client_controller.Read( 'in_inbox', hash )
|
||||
|
||||
if self._file_import_options.ShouldPresent( status, in_inbox ):
|
||||
|
||||
presentation_hashes.append( hash )
|
||||
|
||||
presentation_hashes_fast.add( hash )
|
||||
|
||||
|
||||
|
||||
|
||||
else:
|
||||
|
||||
seed.SetStatus( CC.STATUS_UNINTERESTING_MIME )
|
||||
|
||||
|
||||
except Exception as e:
|
||||
|
||||
error_text = traceback.format_exc()
|
||||
|
||||
HydrusData.Print( 'A file failed to import from import folder ' + self._name + ':' + path )
|
||||
|
||||
seed.SetStatus( CC.STATUS_FAILED, exception = e )
|
||||
|
||||
finally:
|
||||
|
||||
did_work = True
|
||||
|
||||
|
||||
i += 1
|
||||
|
||||
if i % 10 == 0:
|
||||
|
||||
self._ActionPaths()
|
||||
|
||||
|
||||
|
||||
if num_files_imported > 0:
|
||||
|
||||
HydrusData.Print( 'Import folder ' + self._name + ' imported ' + HydrusData.ConvertIntToPrettyString( num_files_imported ) + ' files.' )
|
||||
|
||||
if len( presentation_hashes ) > 0:
|
||||
|
||||
if self._publish_files_to_popup_button:
|
||||
|
||||
job_key = ClientThreading.JobKey()
|
||||
|
||||
job_key.SetVariable( 'popup_files_mergable', True )
|
||||
job_key.SetVariable( 'popup_files', ( list( presentation_hashes ), self._name ) )
|
||||
|
||||
HG.client_controller.pub( 'message', job_key )
|
||||
|
||||
|
||||
if self._publish_files_to_page:
|
||||
|
||||
HG.client_controller.pub( 'imported_files_to_page', list( presentation_hashes ), self._name )
|
||||
|
||||
|
||||
|
||||
|
||||
self._ActionPaths()
|
||||
|
||||
return did_work
|
||||
|
||||
|
||||
def _InitialiseFromSerialisableInfo( self, serialisable_info ):
|
||||
|
||||
( self._path, self._mimes, serialisable_file_import_options, serialisable_tag_import_options, serialisable_tag_service_keys_to_filename_tagging_options, action_pairs, action_location_pairs, self._period, self._open_popup, serialisable_path_cache, self._last_checked, self._paused, self._check_now ) = serialisable_info
|
||||
( self._path, self._mimes, serialisable_file_import_options, serialisable_tag_import_options, serialisable_tag_service_keys_to_filename_tagging_options, action_pairs, action_location_pairs, self._period, self._check_regularly, serialisable_path_cache, self._last_checked, self._paused, self._check_now, self._show_working_popup, self._publish_files_to_popup_button, self._publish_files_to_page ) = serialisable_info
|
||||
|
||||
self._actions = dict( action_pairs )
|
||||
self._action_locations = dict( action_location_pairs )
|
||||
|
|
@ -2271,6 +2499,20 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return ( 5, new_serialisable_info )
|
||||
|
||||
|
||||
if version == 5:
|
||||
|
||||
( path, mimes, serialisable_file_import_options, serialisable_tag_import_options, serialisable_tag_service_keys_to_filename_tagging_options, action_pairs, action_location_pairs, period, open_popup, serialisable_path_cache, last_checked, paused, check_now ) = old_serialisable_info
|
||||
|
||||
check_regularly = not paused
|
||||
show_working_popup = True
|
||||
publish_files_to_page = False
|
||||
publish_files_to_popup_button = open_popup
|
||||
|
||||
new_serialisable_info = ( path, mimes, serialisable_file_import_options, serialisable_tag_import_options, serialisable_tag_service_keys_to_filename_tagging_options, action_pairs, action_location_pairs, period, check_regularly, serialisable_path_cache, last_checked, paused, check_now, show_working_popup, publish_files_to_popup_button, publish_files_to_page )
|
||||
|
||||
return ( 6, new_serialisable_info )
|
||||
|
||||
|
||||
|
||||
def CheckNow( self ):
|
||||
|
||||
|
|
@ -2284,207 +2526,64 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
return
|
||||
|
||||
|
||||
time_to_stop = HydrusData.GetNow() + 600
|
||||
was_interrupted = False
|
||||
if HC.options[ 'pause_import_folders_sync' ] or self._paused:
|
||||
|
||||
return
|
||||
|
||||
|
||||
if not os.path.exists( self._path ) or not os.path.isdir( self._path ):
|
||||
|
||||
return
|
||||
|
||||
|
||||
pubbed_job_key = False
|
||||
|
||||
job_key = ClientThreading.JobKey( pausable = False, cancellable = True )
|
||||
|
||||
job_key.SetVariable( 'popup_title', 'import folder - ' + self._name )
|
||||
|
||||
due_by_check_now = self._check_now
|
||||
due_by_period = not self._paused and HydrusData.TimeHasPassed( self._last_checked + self._period )
|
||||
due_by_period = self._check_regularly and HydrusData.TimeHasPassed( self._last_checked + self._period )
|
||||
|
||||
checked_folder = False
|
||||
|
||||
if due_by_check_now or due_by_period:
|
||||
|
||||
if os.path.exists( self._path ) and os.path.isdir( self._path ):
|
||||
if not pubbed_job_key and self._show_working_popup:
|
||||
|
||||
filenames = os.listdir( HydrusData.ToUnicode( self._path ) )
|
||||
HG.client_controller.pub( 'message', job_key )
|
||||
|
||||
raw_paths = [ os.path.join( self._path, filename ) for filename in filenames ]
|
||||
|
||||
all_paths = ClientFiles.GetAllPaths( raw_paths )
|
||||
|
||||
all_paths = HydrusPaths.FilterFreePaths( all_paths )
|
||||
|
||||
new_paths = []
|
||||
|
||||
for path in all_paths:
|
||||
|
||||
if path.endswith( '.txt' ):
|
||||
|
||||
continue
|
||||
|
||||
|
||||
if not self._path_cache.HasPath( path ):
|
||||
|
||||
new_paths.append( path )
|
||||
|
||||
|
||||
|
||||
self._path_cache.AddPaths( new_paths )
|
||||
|
||||
num_files_imported = 0
|
||||
presentation_hashes = []
|
||||
presentation_hashes_fast = set()
|
||||
|
||||
i = 0
|
||||
|
||||
while True:
|
||||
|
||||
seed = self._path_cache.GetNextSeed( CC.STATUS_UNKNOWN )
|
||||
|
||||
p1 = HC.options[ 'pause_import_folders_sync' ] or self._paused
|
||||
p2 = HG.view_shutdown
|
||||
|
||||
if seed is None or p1 or p2:
|
||||
|
||||
was_interrupted = True
|
||||
|
||||
break
|
||||
|
||||
|
||||
if HydrusData.TimeHasPassed( time_to_stop ):
|
||||
|
||||
was_interrupted = True
|
||||
|
||||
break
|
||||
|
||||
|
||||
path = seed.seed_data
|
||||
|
||||
try:
|
||||
|
||||
mime = HydrusFileHandling.GetMime( path )
|
||||
|
||||
if mime in self._mimes:
|
||||
|
||||
( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
|
||||
|
||||
try:
|
||||
|
||||
copied = HydrusPaths.MirrorFile( path, temp_path )
|
||||
|
||||
if not copied:
|
||||
|
||||
raise Exception( 'File failed to copy--see log for error.' )
|
||||
|
||||
|
||||
file_import_job = FileImportJob( temp_path, self._file_import_options )
|
||||
|
||||
client_files_manager = HG.client_controller.client_files_manager
|
||||
|
||||
( status, hash ) = client_files_manager.ImportFile( file_import_job )
|
||||
|
||||
finally:
|
||||
|
||||
HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
|
||||
|
||||
|
||||
seed.SetStatus( status )
|
||||
|
||||
if status in ( CC.STATUS_SUCCESSFUL, CC.STATUS_REDUNDANT ):
|
||||
|
||||
downloaded_tags = []
|
||||
|
||||
service_keys_to_content_updates = self._tag_import_options.GetServiceKeysToContentUpdates( hash, downloaded_tags ) # explicit tags
|
||||
|
||||
if len( service_keys_to_content_updates ) > 0:
|
||||
|
||||
HG.client_controller.WriteSynchronous( 'content_updates', service_keys_to_content_updates )
|
||||
|
||||
|
||||
service_keys_to_tags = {}
|
||||
|
||||
for ( tag_service_key, filename_tagging_options ) in self._tag_service_keys_to_filename_tagging_options.items():
|
||||
|
||||
if not HG.client_controller.services_manager.ServiceExists( tag_service_key ):
|
||||
|
||||
continue
|
||||
|
||||
|
||||
try:
|
||||
|
||||
tags = filename_tagging_options.GetTags( tag_service_key, path )
|
||||
|
||||
if len( tags ) > 0:
|
||||
|
||||
service_keys_to_tags[ tag_service_key ] = tags
|
||||
|
||||
|
||||
except Exception as e:
|
||||
|
||||
HydrusData.ShowText( 'Trying to parse filename tags in the import folder "' + self._name + '" threw an error!' )
|
||||
|
||||
HydrusData.ShowException( e )
|
||||
|
||||
|
||||
|
||||
if len( service_keys_to_tags ) > 0:
|
||||
|
||||
service_keys_to_content_updates = ClientData.ConvertServiceKeysToTagsToServiceKeysToContentUpdates( { hash }, service_keys_to_tags )
|
||||
|
||||
HG.client_controller.WriteSynchronous( 'content_updates', service_keys_to_content_updates )
|
||||
|
||||
|
||||
num_files_imported += 1
|
||||
|
||||
if hash not in presentation_hashes_fast:
|
||||
|
||||
in_inbox = HG.client_controller.Read( 'in_inbox', hash )
|
||||
|
||||
if self._file_import_options.ShouldPresent( status, in_inbox ):
|
||||
|
||||
presentation_hashes.append( hash )
|
||||
|
||||
presentation_hashes_fast.add( hash )
|
||||
|
||||
|
||||
|
||||
|
||||
else:
|
||||
|
||||
seed.SetStatus( CC.STATUS_UNINTERESTING_MIME )
|
||||
|
||||
|
||||
except Exception as e:
|
||||
|
||||
error_text = traceback.format_exc()
|
||||
|
||||
HydrusData.Print( 'A file failed to import from import folder ' + self._name + ':' )
|
||||
|
||||
seed.SetStatus( CC.STATUS_FAILED, exception = e )
|
||||
|
||||
|
||||
i += 1
|
||||
|
||||
if i % 10 == 0:
|
||||
|
||||
self._ActionPaths()
|
||||
|
||||
|
||||
|
||||
if num_files_imported > 0:
|
||||
|
||||
HydrusData.Print( 'Import folder ' + self._name + ' imported ' + HydrusData.ConvertIntToPrettyString( num_files_imported ) + ' files.' )
|
||||
|
||||
if len( presentation_hashes ) > 0 and self._open_popup:
|
||||
|
||||
job_key = ClientThreading.JobKey()
|
||||
|
||||
job_key.SetVariable( 'popup_title', 'import folder - ' + self._name )
|
||||
job_key.SetVariable( 'popup_files', ( presentation_hashes, self._name ) )
|
||||
|
||||
HG.client_controller.pub( 'message', job_key )
|
||||
|
||||
|
||||
|
||||
self._ActionPaths()
|
||||
pubbed_job_key = True
|
||||
|
||||
|
||||
if not was_interrupted:
|
||||
self._CheckFolder( job_key )
|
||||
|
||||
checked_folder = True
|
||||
|
||||
|
||||
seed = self._path_cache.GetNextSeed( CC.STATUS_UNKNOWN )
|
||||
|
||||
did_import_file_work = False
|
||||
|
||||
if seed is not None:
|
||||
|
||||
if not pubbed_job_key and self._show_working_popup:
|
||||
|
||||
self._last_checked = HydrusData.GetNow()
|
||||
self._check_now = False
|
||||
HG.client_controller.pub( 'message', job_key )
|
||||
|
||||
pubbed_job_key = True
|
||||
|
||||
|
||||
did_import_file_work = self._ImportFiles( job_key )
|
||||
|
||||
|
||||
if checked_folder or did_import_file_work:
|
||||
|
||||
HG.client_controller.WriteSynchronous( 'serialisable', self )
|
||||
|
||||
|
||||
job_key.Delete()
|
||||
|
||||
|
||||
def GetSeedCache( self ):
|
||||
|
||||
|
|
@ -2498,7 +2597,7 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
def ToTuple( self ):
|
||||
|
||||
return ( self._name, self._path, self._mimes, self._file_import_options, self._tag_import_options, self._tag_service_keys_to_filename_tagging_options, self._actions, self._action_locations, self._period, self._open_popup, self._paused, self._check_now )
|
||||
return ( self._name, self._path, self._mimes, self._file_import_options, self._tag_import_options, self._tag_service_keys_to_filename_tagging_options, self._actions, self._action_locations, self._period, self._check_regularly, self._paused, self._check_now, self._show_working_popup, self._publish_files_to_popup_button, self._publish_files_to_page )
|
||||
|
||||
|
||||
def SetSeedCache( self, seed_cache ):
|
||||
|
|
@ -2506,7 +2605,7 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
self._path_cache = seed_cache
|
||||
|
||||
|
||||
def SetTuple( self, name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, open_popup, paused, check_now ):
|
||||
def SetTuple( self, name, path, mimes, file_import_options, tag_import_options, tag_service_keys_to_filename_tagging_options, actions, action_locations, period, check_regularly, paused, check_now, show_working_popup, publish_files_to_popup_button, publish_files_to_page ):
|
||||
|
||||
if path != self._path:
|
||||
|
||||
|
|
@ -2527,9 +2626,12 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
self._actions = actions
|
||||
self._action_locations = action_locations
|
||||
self._period = period
|
||||
self._open_popup = open_popup
|
||||
self._check_regularly = check_regularly
|
||||
self._paused = paused
|
||||
self._check_now = check_now
|
||||
self._show_working_popup = show_working_popup
|
||||
self._publish_files_to_popup_button = publish_files_to_popup_button
|
||||
self._publish_files_to_page = publish_files_to_page
|
||||
|
||||
|
||||
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_IMPORT_FOLDER ] = ImportFolder
|
||||
|
|
@ -4397,6 +4499,7 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
|
|||
|
||||
files_job_key = ClientThreading.JobKey()
|
||||
|
||||
files_job_key.SetVariable( 'popup_files_mergable', True )
|
||||
files_job_key.SetVariable( 'popup_files', ( all_presentation_hashes, file_popup_text ) )
|
||||
|
||||
HG.client_controller.pub( 'message', files_job_key )
|
||||
|
|
@ -5378,14 +5481,17 @@ class ThreadWatcherImport( HydrusSerialisable.SerialisableBase ):
|
|||
watcher_status = 'Could not parse the given URL!'
|
||||
|
||||
|
||||
# convert to API url as appropriate
|
||||
( url_to_check, parser ) = HG.client_controller.network_engine.domain_manager.GetURLToFetchAndParser( self._thread_url )
|
||||
|
||||
if parser is None:
|
||||
if not error_occurred:
|
||||
|
||||
error_occurred = True
|
||||
# convert to API url as appropriate
|
||||
( url_to_check, parser ) = HG.client_controller.network_engine.domain_manager.GetURLToFetchAndParser( self._thread_url )
|
||||
|
||||
watcher_status = 'Could not find a parser for the given URL!'
|
||||
if parser is None:
|
||||
|
||||
error_occurred = True
|
||||
|
||||
watcher_status = 'Could not find a parser for the given URL!'
|
||||
|
||||
|
||||
|
||||
if error_occurred:
|
||||
|
|
|
|||
|
|
@ -249,7 +249,9 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
def key( u_m ):
|
||||
|
||||
return u_m.GetExampleURL().count( '/' )
|
||||
u_e = u_m.GetExampleURL()
|
||||
|
||||
return ( u_e.count( '/' ), u_e.count( '=' ) )
|
||||
|
||||
|
||||
for url_matches in self._domains_to_url_matches.values():
|
||||
|
|
|
|||
|
|
@ -132,7 +132,7 @@ class JobKey( object ):
|
|||
|
||||
else:
|
||||
|
||||
CallLater( HG.client_controller.gui, seconds, self.Cancel )
|
||||
HG.client_controller.CallLater( seconds, self.Cancel )
|
||||
|
||||
|
||||
|
||||
|
|
@ -166,7 +166,7 @@ class JobKey( object ):
|
|||
|
||||
else:
|
||||
|
||||
CallLater( HG.client_controller.gui, seconds, self.Finish )
|
||||
HG.client_controller.CallLater( seconds, self.Finish )
|
||||
|
||||
|
||||
|
||||
|
|
@ -534,3 +534,75 @@ def CallLater( window, seconds, callable, *args, **kwargs ):
|
|||
timer.CallLater( seconds )
|
||||
|
||||
return timer
|
||||
|
||||
class WXAwareJob( HydrusThreading.SchedulableJob ):
|
||||
|
||||
def __init__( self, controller, scheduler, window, work_callable, initial_delay = 0.0 ):
|
||||
|
||||
HydrusThreading.SchedulableJob.__init__( self, controller, scheduler, work_callable, initial_delay = initial_delay )
|
||||
|
||||
self._window = window
|
||||
|
||||
|
||||
def _BootWorker( self ):
|
||||
|
||||
def wx_code():
|
||||
|
||||
if not self._window:
|
||||
|
||||
return
|
||||
|
||||
|
||||
self.Work()
|
||||
|
||||
|
||||
wx.CallAfter( wx_code )
|
||||
|
||||
|
||||
def IsCancelled( self ):
|
||||
|
||||
my_window_dead = not self._window
|
||||
|
||||
if my_window_dead:
|
||||
|
||||
self._is_cancelled.set()
|
||||
|
||||
|
||||
return HydrusThreading.SchedulableJob.IsCancelled( self )
|
||||
|
||||
|
||||
class WXAwareRepeatingJob( HydrusThreading.RepeatingJob ):
|
||||
|
||||
def __init__( self, controller, scheduler, window, work_callable, period, initial_delay = 0.0 ):
|
||||
|
||||
HydrusThreading.RepeatingJob.__init__( self, controller, scheduler, work_callable, period, initial_delay = initial_delay )
|
||||
|
||||
self._window = window
|
||||
|
||||
|
||||
def _BootWorker( self ):
|
||||
|
||||
def wx_code():
|
||||
|
||||
if not self._window:
|
||||
|
||||
return
|
||||
|
||||
|
||||
self.Work()
|
||||
|
||||
|
||||
wx.CallAfter( wx_code )
|
||||
|
||||
|
||||
def IsCancelled( self ):
|
||||
|
||||
my_window_dead = not self._window
|
||||
|
||||
if my_window_dead:
|
||||
|
||||
self._is_cancelled.set()
|
||||
|
||||
|
||||
return HydrusThreading.SchedulableJob.IsCancelled( self )
|
||||
|
||||
|
|
|
|||
|
|
@ -49,7 +49,7 @@ options = {}
|
|||
# Misc
|
||||
|
||||
NETWORK_VERSION = 18
|
||||
SOFTWARE_VERSION = 293
|
||||
SOFTWARE_VERSION = 294
|
||||
|
||||
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )
|
||||
|
||||
|
|
|
|||
|
|
@ -45,9 +45,13 @@ class HydrusController( object ):
|
|||
self._caches = {}
|
||||
self._managers = {}
|
||||
|
||||
self._job_scheduler = None
|
||||
|
||||
self._call_to_threads = []
|
||||
self._long_running_call_to_threads = []
|
||||
|
||||
self._call_to_thread_lock = threading.Lock()
|
||||
|
||||
self._timestamps = collections.defaultdict( lambda: 0 )
|
||||
|
||||
self._timestamps[ 'boot' ] = HydrusData.GetNow()
|
||||
|
|
@ -60,51 +64,57 @@ class HydrusController( object ):
|
|||
|
||||
def _GetCallToThread( self ):
|
||||
|
||||
for call_to_thread in self._call_to_threads:
|
||||
with self._call_to_thread_lock:
|
||||
|
||||
if not call_to_thread.CurrentlyWorking():
|
||||
for call_to_thread in self._call_to_threads:
|
||||
|
||||
return call_to_thread
|
||||
if not call_to_thread.CurrentlyWorking():
|
||||
|
||||
return call_to_thread
|
||||
|
||||
|
||||
|
||||
|
||||
# all the threads in the pool are currently busy
|
||||
|
||||
calling_from_the_thread_pool = threading.current_thread() in self._call_to_threads
|
||||
|
||||
if calling_from_the_thread_pool or len( self._call_to_threads ) < 10:
|
||||
# all the threads in the pool are currently busy
|
||||
|
||||
call_to_thread = HydrusThreading.THREADCallToThread( self )
|
||||
calling_from_the_thread_pool = threading.current_thread() in self._call_to_threads
|
||||
|
||||
self._call_to_threads.append( call_to_thread )
|
||||
if calling_from_the_thread_pool or len( self._call_to_threads ) < 10:
|
||||
|
||||
call_to_thread = HydrusThreading.THREADCallToThread( self, 'CallToThread' )
|
||||
|
||||
self._call_to_threads.append( call_to_thread )
|
||||
|
||||
call_to_thread.start()
|
||||
|
||||
else:
|
||||
|
||||
call_to_thread = random.choice( self._call_to_threads )
|
||||
|
||||
|
||||
call_to_thread.start()
|
||||
return call_to_thread
|
||||
|
||||
else:
|
||||
|
||||
call_to_thread = random.choice( self._call_to_threads )
|
||||
|
||||
|
||||
return call_to_thread
|
||||
|
||||
|
||||
def _GetCallToThreadLongRunning( self ):
|
||||
|
||||
for call_to_thread in self._long_running_call_to_threads:
|
||||
with self._call_to_thread_lock:
|
||||
|
||||
if not call_to_thread.CurrentlyWorking():
|
||||
for call_to_thread in self._long_running_call_to_threads:
|
||||
|
||||
return call_to_thread
|
||||
if not call_to_thread.CurrentlyWorking():
|
||||
|
||||
return call_to_thread
|
||||
|
||||
|
||||
|
||||
|
||||
call_to_thread = HydrusThreading.THREADCallToThread( self )
|
||||
|
||||
self._long_running_call_to_threads.append( call_to_thread )
|
||||
|
||||
call_to_thread.start()
|
||||
|
||||
return call_to_thread
|
||||
call_to_thread = HydrusThreading.THREADCallToThread( self, 'CallToThreadLongRunning' )
|
||||
|
||||
self._long_running_call_to_threads.append( call_to_thread )
|
||||
|
||||
call_to_thread.start()
|
||||
|
||||
return call_to_thread
|
||||
|
||||
|
||||
|
||||
def _InitDB( self ):
|
||||
|
|
@ -112,6 +122,33 @@ class HydrusController( object ):
|
|||
raise NotImplementedError()
|
||||
|
||||
|
||||
def _MaintainCallToThreads( self ):
|
||||
|
||||
# we don't really want to hang on to threads that are done as event.wait() has a bit of idle cpu
|
||||
# so, any that are in the pools that aren't doing anything can be killed and sent to garbage
|
||||
|
||||
with self._call_to_thread_lock:
|
||||
|
||||
def filter_call_to_threads( t ):
|
||||
|
||||
if t.CurrentlyWorking():
|
||||
|
||||
return True
|
||||
|
||||
else:
|
||||
|
||||
t.shutdown()
|
||||
|
||||
return False
|
||||
|
||||
|
||||
|
||||
self._call_to_threads = filter( filter_call_to_threads, self._call_to_threads )
|
||||
|
||||
self._long_running_call_to_threads = filter( filter_call_to_threads, self._long_running_call_to_threads )
|
||||
|
||||
|
||||
|
||||
def _Read( self, action, *args, **kwargs ):
|
||||
|
||||
result = self.db.Read( action, HC.HIGH_PRIORITY, *args, **kwargs )
|
||||
|
|
@ -170,6 +207,17 @@ class HydrusController( object ):
|
|||
self._pubsub.sub( object, method_name, topic )
|
||||
|
||||
|
||||
def CallLater( self, delay, func, *args, **kwargs ):
|
||||
|
||||
call = HydrusData.Call( func, *args, **kwargs )
|
||||
|
||||
job = HydrusThreading.SchedulableJob( self, self._job_scheduler, call, initial_delay = delay )
|
||||
|
||||
self._job_scheduler.AddJob( job )
|
||||
|
||||
return job
|
||||
|
||||
|
||||
def CallToThread( self, callable, *args, **kwargs ):
|
||||
|
||||
if HG.callto_report_mode:
|
||||
|
|
@ -294,6 +342,10 @@ class HydrusController( object ):
|
|||
|
||||
self.temp_dir = HydrusPaths.GetTempDir()
|
||||
|
||||
self._job_scheduler = HydrusThreading.JobScheduler( self )
|
||||
|
||||
self._job_scheduler.start()
|
||||
|
||||
self.db = self._InitDB()
|
||||
|
||||
|
||||
|
|
@ -335,6 +387,8 @@ class HydrusController( object ):
|
|||
|
||||
HydrusPaths.CleanUpOldTempPaths()
|
||||
|
||||
self._MaintainCallToThreads()
|
||||
|
||||
|
||||
def ModelIsShutdown( self ):
|
||||
|
||||
|
|
@ -392,6 +446,13 @@ class HydrusController( object ):
|
|||
|
||||
|
||||
|
||||
if self._job_scheduler is not None:
|
||||
|
||||
self._job_scheduler.shutdown()
|
||||
|
||||
self._job_scheduler = None
|
||||
|
||||
|
||||
if hasattr( self, 'temp_dir' ):
|
||||
|
||||
HydrusPaths.DeletePath( self.temp_dir )
|
||||
|
|
|
|||
|
|
@ -344,6 +344,15 @@ class HydrusDB( object ):
|
|||
self._c.execute( statement )
|
||||
|
||||
|
||||
def _DisplayCatastrophicError( self, text ):
|
||||
|
||||
message = 'The db encountered a serious error! This is going to be written to the log as well, but here it is for a screenshot:'
|
||||
message += os.linesep * 2
|
||||
message += text
|
||||
|
||||
HydrusData.DebugPrint( message )
|
||||
|
||||
|
||||
def _GetRowCount( self ):
|
||||
|
||||
row_count = self._c.rowcount
|
||||
|
|
@ -753,7 +762,7 @@ class HydrusDB( object ):
|
|||
|
||||
except:
|
||||
|
||||
HydrusData.Print( traceback.format_exc() )
|
||||
self._DisplayCatastrophicError( traceback.format_exc() )
|
||||
|
||||
self._could_not_initialise = True
|
||||
|
||||
|
|
|
|||
|
|
@ -528,10 +528,12 @@ def ConvertTimestampToPrettyPending( timestamp, prefix = 'in' ):
|
|||
if timestamp is None: return ''
|
||||
if timestamp == 0: return 'imminent'
|
||||
|
||||
pending = GetNow() - timestamp
|
||||
pending = GetTimeDeltaUntilTime( timestamp )
|
||||
|
||||
if pending >= 0: return 'imminent'
|
||||
else: pending *= -1
|
||||
if pending <= 0:
|
||||
|
||||
return 'imminent'
|
||||
|
||||
|
||||
seconds = pending % 60
|
||||
if seconds == 1: s = '1 second'
|
||||
|
|
@ -772,6 +774,10 @@ def GetNow():
|
|||
|
||||
return int( time.time() )
|
||||
|
||||
def GetNowFloat():
|
||||
|
||||
return time.time()
|
||||
|
||||
def GetNowPrecise():
|
||||
|
||||
if HC.PLATFORM_WINDOWS:
|
||||
|
|
@ -833,6 +839,24 @@ def GetSiblingProcessPorts( db_path, instance ):
|
|||
|
||||
return None
|
||||
|
||||
def GetTimeDeltaUntilTime( timestamp ):
|
||||
|
||||
time_remaining = timestamp - GetNow()
|
||||
|
||||
return max( time_remaining, 0 )
|
||||
|
||||
def GetTimeDeltaUntilTimeFloat( timestamp ):
|
||||
|
||||
time_remaining = timestamp - GetNowFloat()
|
||||
|
||||
return max( time_remaining, 0.0 )
|
||||
|
||||
def GetTimeDeltaUntilTimePrecise( t ):
|
||||
|
||||
time_remaining = t - GetNowPrecise()
|
||||
|
||||
return max( time_remaining, 0.0 )
|
||||
|
||||
def IntelligentMassIntersect( sets_to_reduce ):
|
||||
|
||||
answer = None
|
||||
|
|
@ -1195,6 +1219,10 @@ def TimeHasPassed( timestamp ):
|
|||
|
||||
return GetNow() > timestamp
|
||||
|
||||
def TimeHasPassedFloat( timestamp ):
|
||||
|
||||
return GetNowFloat() > timestamp
|
||||
|
||||
def TimeHasPassedPrecise( precise_timestamp ):
|
||||
|
||||
return GetNowPrecise() > precise_timestamp
|
||||
|
|
@ -1636,6 +1664,11 @@ class Call( object ):
|
|||
self._func( *self._args, **self._kwargs )
|
||||
|
||||
|
||||
def __repr__( self ):
|
||||
|
||||
return 'Call: ' + repr( ( self._func, self._args, self._kwargs ) )
|
||||
|
||||
|
||||
class ContentUpdate( object ):
|
||||
|
||||
def __init__( self, data_type, action, row ):
|
||||
|
|
|
|||
|
|
@ -639,7 +639,7 @@ class BandwidthTracker( HydrusSerialisable.SerialisableBase ):
|
|||
|
||||
next_month_time = int( calendar.timegm( next_month_dt.timetuple() ) )
|
||||
|
||||
return next_month_time - HydrusData.GetNow()
|
||||
return HydrusData.GetTimeDeltaUntilTime( next_month_time )
|
||||
|
||||
else:
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import bisect
|
||||
import collections
|
||||
import HydrusExceptions
|
||||
import Queue
|
||||
|
|
@ -51,7 +52,7 @@ def ShutdownThread( thread ):
|
|||
|
||||
class DAEMON( threading.Thread ):
|
||||
|
||||
def __init__( self, controller, name, period = 1200 ):
|
||||
def __init__( self, controller, name ):
|
||||
|
||||
threading.Thread.__init__( self, name = name )
|
||||
|
||||
|
|
@ -193,266 +194,11 @@ class DAEMONForegroundWorker( DAEMONWorker ):
|
|||
return self._controller.GoodTimeToDoForegroundWork()
|
||||
|
||||
|
||||
class JobScheduler( DAEMON ):
|
||||
|
||||
def __init__( self, controller ):
|
||||
|
||||
DAEMON.__init__( self, controller, 'JobScheduler' )
|
||||
|
||||
self._currently_working = []
|
||||
|
||||
self._waiting = []
|
||||
|
||||
self._waiting_lock = threading.Lock()
|
||||
|
||||
self._new_action = threading.Event()
|
||||
|
||||
self._sort_needed = threading.Event()
|
||||
|
||||
|
||||
def _InsertJob( self, job ):
|
||||
|
||||
# write __lt__, __gt__, stuff and do a bisect insort_left here
|
||||
|
||||
with self._waiting_lock:
|
||||
|
||||
self._waiting.append( job )
|
||||
|
||||
|
||||
self._sort_needed.set()
|
||||
|
||||
|
||||
def _NoWorkToStart( self ):
|
||||
|
||||
with self._waiting_lock:
|
||||
|
||||
if len( self._waiting ) == 0:
|
||||
|
||||
return True
|
||||
|
||||
|
||||
next_job = self._waiting[0]
|
||||
|
||||
|
||||
if HydrusData.TimeHasPassed( next_job.GetNextWorkTime() ):
|
||||
|
||||
return False
|
||||
|
||||
else:
|
||||
|
||||
return True
|
||||
|
||||
|
||||
|
||||
def _RescheduleFinishedJobs( self ):
|
||||
|
||||
def reschedule_finished_job( job ):
|
||||
|
||||
if job.CurrentlyWorking():
|
||||
|
||||
return True
|
||||
|
||||
else:
|
||||
|
||||
self._InsertJob( job )
|
||||
|
||||
return False
|
||||
|
||||
|
||||
|
||||
self._currently_working = filter( reschedule_finished_job, self._currently_working )
|
||||
|
||||
|
||||
def _SortWaiting( self ):
|
||||
|
||||
# sort the waiting jobs in ascending order of expected work time
|
||||
|
||||
def key( job ):
|
||||
|
||||
return job.GetNextWorkTime()
|
||||
|
||||
|
||||
with self._waiting_lock:
|
||||
|
||||
self._waiting.sort( key = key )
|
||||
|
||||
|
||||
|
||||
def _StartWork( self ):
|
||||
|
||||
while True:
|
||||
|
||||
with self._waiting_lock:
|
||||
|
||||
if len( self._waiting ) == 0:
|
||||
|
||||
break
|
||||
|
||||
|
||||
next_job = self._waiting[0]
|
||||
|
||||
if HydrusData.TimeHasPassed( next_job.GetNextWorkTime() ):
|
||||
|
||||
next_job = self._waiting.pop( 0 )
|
||||
|
||||
if not next_job.IsDead():
|
||||
|
||||
next_job.StartWork()
|
||||
|
||||
self._currently_working.append( next_job )
|
||||
|
||||
|
||||
else:
|
||||
|
||||
break # all the rest in the queue are not due
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
def RegisterJob( self, job ):
|
||||
|
||||
job.SetScheduler( self )
|
||||
|
||||
self._InsertJob( job )
|
||||
|
||||
|
||||
def WorkTimesHaveChanged( self ):
|
||||
|
||||
self._sort_needed.set()
|
||||
|
||||
|
||||
def run( self ):
|
||||
|
||||
while True:
|
||||
|
||||
try:
|
||||
|
||||
while self._NoWorkToStart():
|
||||
|
||||
if self._controller.ModelIsShutdown():
|
||||
|
||||
return
|
||||
|
||||
|
||||
#
|
||||
|
||||
self._RescheduleFinishedJobs()
|
||||
|
||||
#
|
||||
|
||||
self._sort_needed.wait( 0.2 )
|
||||
|
||||
if self._sort_needed.is_set():
|
||||
|
||||
self._SortWaiting()
|
||||
|
||||
self._sort_needed.clear()
|
||||
|
||||
|
||||
|
||||
self._StartWork()
|
||||
|
||||
except HydrusExceptions.ShutdownException:
|
||||
|
||||
return
|
||||
|
||||
except Exception as e:
|
||||
|
||||
HydrusData.Print( traceback.format_exc() )
|
||||
|
||||
HydrusData.ShowException( e )
|
||||
|
||||
|
||||
time.sleep( 0.00001 )
|
||||
|
||||
|
||||
|
||||
class RepeatingJob( object ):
|
||||
|
||||
def __init__( self, controller, work_callable, period, initial_delay = 0 ):
|
||||
|
||||
self._controller = controller
|
||||
self._work_callable = work_callable
|
||||
self._period = period
|
||||
|
||||
self._is_dead = threading.Event()
|
||||
|
||||
self._work_lock = threading.Lock()
|
||||
|
||||
self._currently_working = threading.Event()
|
||||
|
||||
self._next_work_time = HydrusData.GetNow() + initial_delay
|
||||
|
||||
self._scheduler = None
|
||||
|
||||
# registers itself with controller here
|
||||
|
||||
|
||||
def CurrentlyWorking( self ):
|
||||
|
||||
return self._currently_working.is_set()
|
||||
|
||||
|
||||
def GetNextWorkTime( self ):
|
||||
|
||||
return self._next_work_time
|
||||
|
||||
|
||||
def IsDead( self ):
|
||||
|
||||
return self._is_dead.is_set()
|
||||
|
||||
|
||||
def Kill( self ):
|
||||
|
||||
self._is_dead.set()
|
||||
|
||||
|
||||
def SetScheduler( self, scheduler ):
|
||||
|
||||
self._scheduler = scheduler
|
||||
|
||||
|
||||
def StartWork( self ):
|
||||
|
||||
self._currently_working.set()
|
||||
|
||||
self._controller.CallToThread( self.Work )
|
||||
|
||||
|
||||
def WakeAndWork( self ):
|
||||
|
||||
self._next_work_time = HydrusData.GetNow()
|
||||
|
||||
if self._scheduler is not None:
|
||||
|
||||
self._scheduler.WorkTimesHaveChanged()
|
||||
|
||||
|
||||
|
||||
def Work( self ):
|
||||
|
||||
with self._work_lock:
|
||||
|
||||
try:
|
||||
|
||||
self._work_callable()
|
||||
|
||||
finally:
|
||||
|
||||
self._next_work_time = HydrusData.GetNow() + self._period
|
||||
|
||||
self._currently_working.clear()
|
||||
|
||||
|
||||
|
||||
|
||||
class THREADCallToThread( DAEMON ):
|
||||
|
||||
def __init__( self, controller ):
|
||||
def __init__( self, controller, name ):
|
||||
|
||||
DAEMON.__init__( self, controller, 'CallToThread' )
|
||||
DAEMON.__init__( self, controller, name )
|
||||
|
||||
self._queue = Queue.Queue()
|
||||
|
||||
|
|
@ -518,3 +264,338 @@ class THREADCallToThread( DAEMON ):
|
|||
|
||||
|
||||
|
||||
class JobScheduler( threading.Thread ):
|
||||
|
||||
def __init__( self, controller ):
|
||||
|
||||
threading.Thread.__init__( self, name = 'Job Scheduler' )
|
||||
|
||||
self._controller = controller
|
||||
|
||||
self._waiting = []
|
||||
|
||||
self._waiting_lock = threading.Lock()
|
||||
|
||||
self._new_job_arrived = threading.Event()
|
||||
|
||||
self._cancel_filter_needed = threading.Event()
|
||||
self._sort_needed = threading.Event()
|
||||
|
||||
self._controller.sub( self, 'shutdown', 'shutdown' )
|
||||
|
||||
|
||||
def _FilterCancelled( self ):
|
||||
|
||||
with self._waiting_lock:
|
||||
|
||||
self._waiting = [ job for job in self._waiting if not job.IsCancelled() ]
|
||||
|
||||
|
||||
|
||||
def _GetLoopWaitTime( self ):
|
||||
|
||||
with self._waiting_lock:
|
||||
|
||||
if len( self._waiting ) == 0:
|
||||
|
||||
return 0.2
|
||||
|
||||
|
||||
next_job = self._waiting[0]
|
||||
|
||||
|
||||
time_delta_until_due = next_job.GetTimeDeltaUntilDue()
|
||||
|
||||
return min( 1.0, time_delta_until_due )
|
||||
|
||||
|
||||
def _NoWorkToStart( self ):
|
||||
|
||||
with self._waiting_lock:
|
||||
|
||||
if len( self._waiting ) == 0:
|
||||
|
||||
return True
|
||||
|
||||
|
||||
next_job = self._waiting[0]
|
||||
|
||||
|
||||
if next_job.IsDue():
|
||||
|
||||
return False
|
||||
|
||||
else:
|
||||
|
||||
return True
|
||||
|
||||
|
||||
|
||||
def _SortWaiting( self ):
|
||||
|
||||
# sort the waiting jobs in ascending order of expected work time
|
||||
|
||||
with self._waiting_lock: # this uses __lt__ to sort
|
||||
|
||||
self._waiting.sort()
|
||||
|
||||
|
||||
|
||||
def _StartWork( self ):
|
||||
|
||||
while True:
|
||||
|
||||
with self._waiting_lock:
|
||||
|
||||
if len( self._waiting ) == 0:
|
||||
|
||||
break
|
||||
|
||||
|
||||
next_job = self._waiting[0]
|
||||
|
||||
if next_job.IsDue():
|
||||
|
||||
next_job = self._waiting.pop( 0 )
|
||||
|
||||
next_job.StartWork()
|
||||
|
||||
else:
|
||||
|
||||
break # all the rest in the queue are not due
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
def AddJob( self, job ):
|
||||
|
||||
with self._waiting_lock:
|
||||
|
||||
bisect.insort( self._waiting, job )
|
||||
|
||||
|
||||
self._new_job_arrived.set()
|
||||
|
||||
|
||||
def JobCancelled( self ):
|
||||
|
||||
self._cancel_filter_needed.set()
|
||||
|
||||
|
||||
def shutdown( self ):
|
||||
|
||||
ShutdownThread( self )
|
||||
|
||||
|
||||
def WorkTimesHaveChanged( self ):
|
||||
|
||||
self._sort_needed.set()
|
||||
|
||||
|
||||
def run( self ):
|
||||
|
||||
while True:
|
||||
|
||||
try:
|
||||
|
||||
while self._NoWorkToStart():
|
||||
|
||||
if self._controller.ModelIsShutdown():
|
||||
|
||||
return
|
||||
|
||||
|
||||
#
|
||||
|
||||
if self._cancel_filter_needed.is_set():
|
||||
|
||||
self._FilterCancelled()
|
||||
|
||||
self._cancel_filter_needed.clear()
|
||||
|
||||
|
||||
if self._sort_needed.is_set():
|
||||
|
||||
self._SortWaiting()
|
||||
|
||||
self._sort_needed.clear()
|
||||
|
||||
continue # if some work is now due, let's do it!
|
||||
|
||||
|
||||
#
|
||||
|
||||
wait_time = self._GetLoopWaitTime()
|
||||
|
||||
self._new_job_arrived.wait( wait_time )
|
||||
|
||||
self._new_job_arrived.clear()
|
||||
|
||||
|
||||
self._StartWork()
|
||||
|
||||
except HydrusExceptions.ShutdownException:
|
||||
|
||||
return
|
||||
|
||||
except Exception as e:
|
||||
|
||||
HydrusData.Print( traceback.format_exc() )
|
||||
|
||||
HydrusData.ShowException( e )
|
||||
|
||||
|
||||
time.sleep( 0.00001 )
|
||||
|
||||
|
||||
|
||||
class SchedulableJob( object ):
|
||||
|
||||
def __init__( self, controller, scheduler, work_callable, initial_delay = 0.0 ):
|
||||
|
||||
self._controller = controller
|
||||
self._scheduler = scheduler
|
||||
self._work_callable = work_callable
|
||||
|
||||
self._next_work_time = HydrusData.GetNowFloat() + initial_delay
|
||||
|
||||
self._work_lock = threading.Lock()
|
||||
|
||||
self._currently_working = threading.Event()
|
||||
self._is_cancelled = threading.Event()
|
||||
|
||||
|
||||
def __lt__( self, other ): # for the scheduler to do bisect.insort noice
|
||||
|
||||
return self._next_work_time < other._next_work_time
|
||||
|
||||
|
||||
def __repr__( self ):
|
||||
|
||||
return 'Schedulable Job: ' + repr( self._work_callable )
|
||||
|
||||
|
||||
def _BootWorker( self ):
|
||||
|
||||
self._controller.CallToThread( self.Work )
|
||||
|
||||
|
||||
def Cancel( self ):
|
||||
|
||||
self._is_cancelled.set()
|
||||
|
||||
self._scheduler.JobCancelled()
|
||||
|
||||
|
||||
def CurrentlyWorking( self ):
|
||||
|
||||
return self._currently_working.is_set()
|
||||
|
||||
|
||||
def GetTimeDeltaUntilDue( self ):
|
||||
|
||||
return HydrusData.GetTimeDeltaUntilTimeFloat( self._next_work_time )
|
||||
|
||||
|
||||
def IsCancelled( self ):
|
||||
|
||||
return self._is_cancelled.is_set()
|
||||
|
||||
|
||||
def IsDue( self ):
|
||||
|
||||
return HydrusData.TimeHasPassedFloat( self._next_work_time )
|
||||
|
||||
|
||||
def MoveNextWorkTimeToNow( self ):
|
||||
|
||||
self._next_work_time = HydrusData.GetNowFloat()
|
||||
|
||||
self._scheduler.WorkTimesHaveChanged()
|
||||
|
||||
|
||||
def StartWork( self ):
|
||||
|
||||
if self._is_cancelled.is_set():
|
||||
|
||||
return
|
||||
|
||||
|
||||
self._currently_working.set()
|
||||
|
||||
self._BootWorker()
|
||||
|
||||
|
||||
def Work( self ):
|
||||
|
||||
try:
|
||||
|
||||
with self._work_lock:
|
||||
|
||||
self._work_callable()
|
||||
|
||||
|
||||
finally:
|
||||
|
||||
self._currently_working.clear()
|
||||
|
||||
|
||||
|
||||
class RepeatingJob( SchedulableJob ):
|
||||
|
||||
def __init__( self, controller, scheduler, work_callable, period, initial_delay = 0.0 ):
|
||||
|
||||
SchedulableJob.__init__( self, controller, scheduler, work_callable, initial_delay = initial_delay )
|
||||
|
||||
self._period = period
|
||||
|
||||
self._stop_repeating = threading.Event()
|
||||
|
||||
|
||||
def Cancel( self ):
|
||||
|
||||
SchedulableJob.Cancel( self )
|
||||
|
||||
self._stop_repeating.set()
|
||||
|
||||
|
||||
def Delay( self, delay ):
|
||||
|
||||
self._next_work_time = HydrusData.GetNowFloat() + delay
|
||||
|
||||
self._scheduler.WorkTimesHaveChanged()
|
||||
|
||||
|
||||
def IsFinishedWorking( self ):
|
||||
|
||||
return self._stop_repeating.is_set()
|
||||
|
||||
|
||||
def SetPeriod( self, period ):
|
||||
|
||||
self._period = period
|
||||
|
||||
|
||||
def StartWork( self ):
|
||||
|
||||
if self._stop_repeating.is_set():
|
||||
|
||||
return
|
||||
|
||||
|
||||
SchedulableJob.StartWork( self )
|
||||
|
||||
|
||||
def Work( self ):
|
||||
|
||||
SchedulableJob.Work( self )
|
||||
|
||||
if not self._stop_repeating.is_set():
|
||||
|
||||
self._next_work_time = HydrusData.GetNowFloat() + self._period
|
||||
|
||||
self._scheduler.AddJob( self )
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -668,11 +668,11 @@ class VideoRendererFFMPEG( object ):
|
|||
|
||||
if start_index == 0:
|
||||
|
||||
do_ss = True
|
||||
do_ss = False
|
||||
|
||||
else:
|
||||
|
||||
do_ss = False
|
||||
do_ss = True
|
||||
|
||||
|
||||
ss = float( start_index ) / self.fps
|
||||
|
|
|
|||
|
|
@ -834,8 +834,8 @@ class TestClientDB( unittest.TestCase ):
|
|||
|
||||
def test_import_folders( self ):
|
||||
|
||||
import_folder_1 = ClientImporting.ImportFolder( 'imp 1', path = TestConstants.DB_DIR, mimes = HC.VIDEO, open_popup = False )
|
||||
import_folder_2 = ClientImporting.ImportFolder( 'imp 2', path = TestConstants.DB_DIR, mimes = HC.IMAGES, period = 1200, open_popup = False )
|
||||
import_folder_1 = ClientImporting.ImportFolder( 'imp 1', path = TestConstants.DB_DIR, mimes = HC.VIDEO, publish_files_to_popup_button = False )
|
||||
import_folder_2 = ClientImporting.ImportFolder( 'imp 2', path = TestConstants.DB_DIR, mimes = HC.IMAGES, period = 1200, publish_files_to_popup_button = False )
|
||||
|
||||
#
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue