46 lines
5.9 KiB
HTML
46 lines
5.9 KiB
HTML
<html>
|
|
<head>
|
|
<title>downloader - intro</title>
|
|
<link href="hydrus.ico" rel="shortcut icon" />
|
|
<link href="style.css" rel="stylesheet" type="text/css" />
|
|
</head>
|
|
<body>
|
|
<div class="content">
|
|
<p class="warning">Creating custom downloaders is only for advanced users who understand HTML or JSON. Beware! If you are simply looking for how to add new downloaders, please head over <a href="adding_new_downloaders.html">here</a>.</p>
|
|
<h3 id="intro"><a href="#intro">this system</a></h3>
|
|
<p>The first versions of hydrus's downloaders were all hardcoded and static--I wrote everything into the program itself and nothing was user-creatable or -fixable. After the maintenance burden of the entire messy system proved too large for me to keep up with and a semi-editable booru system proved successful, I decided to overhaul the entire thing to allow user creation and sharing of every component. It is designed to be very simple to the front-end user--they will typically handle a couple of png files and then select a new downloader from a list--but very flexible (and hence potentially complicated) on the back-end. These help pages describe the different compontents with the intention of making an HTML- or JSON- fluent user able to create and share a full new downloader on their own.</p>
|
|
<p>As always, this is all under active development. Your feedback on the system would be appreciated, and if something is confusing or you discover something in here that is out of date, please <a href="contact.html">let me know</a>.</p>
|
|
<h3 id="downloader"><a href="#downloader">what is a downloader?</a></h3>
|
|
<p>In hydrus, a downloader is one of:</p>
|
|
<ul>
|
|
<li><h4>Gallery Downloader</h4></li>
|
|
<li>This takes a string like 'blue_eyes' to produce a series of thumbnail gallery page URLs that can be parsed for image page URLs which can ultimately be parsed for file URLs and metadata like tags. Boorus fall into this category.</li>
|
|
<li><h4>URL Downloader</h4></li>
|
|
<li>This does just the Gallery Downloader's back-end--instead of taking a string query, it takes the gallery or post URLs directly from the user, whether that is one from a drag-and-drop event or hundreds pasted from clipboard. For our purposes here, the URL Downloader is a subset of the Gallery Downloader.</li>
|
|
<li><h4>Watcher</h4></li>
|
|
<li>This takes a URL that it will check in timed intervals, parsing it for new URLs that it then queues up to be downloaded. It typically stops checking after the 'file velocity' (such as '1 new file per day') drops below a certain level. It is mostly for watching imageboard threads.</li>
|
|
<li><h4>Simple Downloader</h4></li>
|
|
<li>This takes a URL one-time and parses it for direct file URLs. This is a miscellaneous system for certain simple gallery types and some testing/'I just need the third <img> tag's <i>src</i> on this one page' jobs.</li>
|
|
</ul>
|
|
<p>The system currently supports HTML and JSON parsing. XML should be fine under the HTML parser--it isn't strict about checking types and all that.</p>
|
|
<h3 id="pipeline"><a href="#pipeline">what does a downloader do?</a></h3>
|
|
<p>The Gallery Downloader is the most complicated downloader and uses all the possible components. In order for hydrus to convert our example 'blue_eyes' query into a bunch of files with tags, it needs to:</p>
|
|
<ul>
|
|
<li>Present some user interface named 'safebooru tag search' to the user that will convert their input of 'blue_eyes' into <a href="https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0">https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0</a>.</li>
|
|
<li>Recognise <a href="https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0">https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=0</a> as a Safebooru Gallery URL.</li>
|
|
<li>Convert the HTML of a Safebooru Gallery URL into a list URLs like <a href="https://safebooru.org/index.php?page=post&s=view&id=2437965">https://safebooru.org/index.php?page=post&s=view&id=2437965</a> and possibly a 'next page' URL (e.g. <a href="https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=40">https://safebooru.org/index.php?page=post&s=list&tags=blue_eyes&pid=40</a>) that points to the next page of thumbnails.</li>
|
|
<li>Recognise the <a href="https://safebooru.org/index.php?page=post&s=view&id=2437965">https://safebooru.org/index.php?page=post&s=view&id=2437965</a> URLs as Safebooru Post URLs.</li>
|
|
<li>Convert the HTML of a Safebooru Post URL into a file URL like <a href="https://safebooru.org//images/2329/b6e8c263d691d1c39a2eeba5e00709849d8f864d.jpg">https://safebooru.org//images/2329/b6e8c263d691d1c39a2eeba5e00709849d8f864d.jpg</a> and some tags like: 1girl, bangs, black gloves, blonde hair, blue eyes, braid, closed mouth, day, fingerless gloves, fingernails, gloves, grass, hair ornament, hairclip, hands clasped, creator:hankuri, interlocked fingers, long hair, long sleeves, outdoors, own hands together, parted bangs, pointy ears, character:princess zelda, smile, solo, series:the legend of zelda, underbust.</li>
|
|
</ul>
|
|
<p>So we have three components:</p>
|
|
<ul>
|
|
<li><a href="downloader_gugs.html"><b>Gallery URL Generator (GUG):</b></a> faces the user and converts text input into initialising Gallery URLs.</li>
|
|
<li><a href="downloader_url_classes.html"><b>URL Class:</b></a> identifies URLs and informs the client how to deal with them.</li>
|
|
<li><a href="downloader_parsers.html"><b>Parser:</b></a> converts data from URLs into hydrus-understandable metadata.</li>
|
|
</ul>
|
|
<p>URL downloaders and watchers do not need the Gallery URL Generator, as their input <i>is</i> an URL. And simple downloaders also have an explicit 'just download it and parse it with this simple rule' action, so they do not use URL Classes (or even full-fledged Page Parsers) either.</p>
|
|
<p class="right"><a href="downloader_gugs.html">Let's first learn about GUGs ----></a></p>
|
|
</div>
|
|
</body>
|
|
</html>
|