changelog
-
+
version 427
+ - ghost pending tags: +
- fixed another ghost pending tags bug. this may have been new or there since the display cache started, I am not sure, but it shouldn't happen again. it was occuring when a pending tag was being committed to 'current' and another tag in its sibling group already existed as a current tag for that file. the pending tag and its count would not clear for non-'all known files' domains, causing ghosts to appear in search pages but not typically manage tags. you may have noticed ghost tags hanging around after a pending commit--this was it. the true problem here was in the 'rescind pending' action that occurs just before an add/commit. a new unit test tests for this situation both for two non-ideal tags being pend-merged, and a non-ideal tag being pend-merged into the existing ideal +
- wrote a routine to regenerate _pending_ tag storage and autocomplete counts from scratch for the combined and specific display tag caches. this job is special in that it regens tags instantly without having to reset sibling/parent sync. you can run the job from the database->regen menu. this is the start of 'fix just this tag' maintenance ability +
- the pending regen routines will occur on update. it shouldn't take long at all, unless you have five million tags pending, where it could be a couple minutes +
- . +
- autocomplete shortcuts: +
- there is a new shortcut set under _file->shortcuts_ just for tag autocomplete shortcuts. any 'switch searching immediately' shortcut previously on 'main gui' will be migrated over +
- the tag autocomplete input text box is now plugged into the new shortcut system and uses this set +
- migrated previously hardcoded autocomplete shortcuts to the shortcut system (defaults): +
- - force search now, for when you have automatic searching turned off (ctrl+space) +
- - enable IME-friendly mode (insert) +
- - if input empty, move left/right a tab (left/right arrow) +
- - if input empty, move left/right a service page (up/down arrow) +
- - if input empty and on media viewer, move to previous/next media (page up/down) +
- misc improvements to my shortcut handler +
- misc shortcut code cleanup +
- . +
- the rest: +
- I fixed a bad example url in the new gelbooru file page parser that was sometimes leading to a link to the gallery url class. this was an artifact of an old experiment with md5-search parsing, now fixed with newer redirection tech. the updated parser is folded into update, and if you ended up with the incorrect link, it should be detected, dissolved, and re-linked with the file page parser +
- thanks to a user report, wrote a new url class for 420chan's newer thread url format +
- sorting a gallery downloader or thread watcher multi-column list by 'status' should now group 'done' and 'paused' items separately +
- fixed a bug in the /add_tags/add_tags Client API call when checking some petitioned tags data types. cleaned all that code, it was ugly (issue #788) +
- added unit tests for /add_tags/add_tags to test the service_names_to_actions_to_tags parameter better and repository actions, including petitioning with and without specified reason +
- . +
- code refactoring: +
- finally addressing the near-1MB ClientDB file, I have started a framework to break the db into separate modules with their own creation/repair/work responsibilities. this will make the file easier to work on, maintain, update, and test. this week starts off simple, with the master definitions being peeled off into hashes, tags, urls, and texts submodules +
- cleaned some misc code around here, including a bunch of related decoupling +
- ClientDB.py is now in its own 'db' module as well. the db will further fracture and this module will gain more files in future +
- the boot code in the launch scripts is now migrated to the 'hydrus' directory, with the actual launch scripts now doing nicer __main__ checks to not launch the program if you want to play around with importing hydrus. more work to come here +
- finished the help's header linking job--all headers across the help are now #fragment links +
- misc help cleanup +
version 426
- misc: @@ -36,7 +70,7 @@
- the parser 'show what this can parse in nice text' routine now fails gracefully
- multi-column lists now handle a situation where either the display or sort data for a row cannot be generated. a single error popup per list will be generated so as not to spam, bad sorts will be put at the top, and 'unable to render' will occupy all display cells
- . -
- network server stuff +
- network server stuff:
- fixed being able to delete an account type in the server admin menu
- the way accounts are checked for permissions serverside now works how the client api does it, unified into a neater system that checks before the job starts
- did some misc server code cleanup, and clientside, prepped for restoring account modification and future improvements diff --git a/help/client_api.html b/help/client_api.html index 06479c3e..3a9cb4be 100644 --- a/help/client_api.html +++ b/help/client_api.html @@ -43,17 +43,17 @@
-
-
Access Management
+Access Management
-Adding Files
+Adding Files
- POST /add_files/add_file
- POST /add_files/delete_files @@ -61,31 +61,31 @@
- POST /add_files/archive_files
- POST /add_files/unarchive_files
Adding Tags
+Adding Tags
-Adding URLs
+Adding URLs
- GET /add_urls/get_url_files
- GET /add_urls/get_url_info
- POST /add_urls/add_url
- POST /add_urls/associate_url
Managing Cookies
+Managing Cookies
-Managing Pages
+Managing Pages
-Searching and Fetching Files
+Searching and Fetching Files
- GET /get_files/search_files
- GET /get_files/file_metadata @@ -93,9 +93,9 @@
- GET /get_files/thumbnail
Restricted access: NO.
@@ -110,8 +110,8 @@
Restricted access: NO.
@@ -149,8 +149,8 @@
Restricted access: YES. No permissions required.
@@ -171,8 +171,8 @@
Restricted access: YES. No permissions required.
@@ -192,9 +192,9 @@
Restricted access: YES. Import Files permission needed.
@@ -232,8 +232,8 @@
Restricted access: YES. Import Files permission needed.
@@ -259,8 +259,8 @@
Restricted access: YES. Import Files permission needed.
@@ -286,8 +286,8 @@
Restricted access: YES. Import Files permission needed.
@@ -313,8 +313,8 @@
Restricted access: YES. Import Files permission needed.
@@ -340,9 +340,9 @@
Restricted access: YES. Add Tags permission needed.
@@ -374,8 +374,8 @@
Restricted access: YES. Add Tags permission needed.
@@ -398,8 +398,8 @@
Restricted access: YES. Add Tags permission needed.
@@ -462,9 +462,9 @@
Restricted access: YES. Import URLs permission needed.
@@ -511,8 +511,8 @@
Restricted access: YES. Import URLs permission needed.
@@ -557,8 +557,8 @@
Restricted access: YES. Import URLs permission needed. Add Tags needed to include tags.
@@ -650,8 +650,8 @@
Restricted access: YES. Import URLs permission needed.
@@ -686,10 +686,11 @@
Response description: 200 with no content. Like when adding tags, this is safely idempotent--do not worry about re-adding URLs associations that already exist or accidentally trying to delete ones that don't.
Restricted access: YES. Manage Cookies permission needed.
@@ -722,8 +723,8 @@
Restricted access: YES. Manage Cookies permission needed.
@@ -756,10 +757,10 @@
Restricted access: YES. Manage Pages permission needed.
@@ -833,8 +834,8 @@
Restricted access: YES. Manage Pages permission needed.
@@ -960,10 +961,10 @@
Response description: 200 with no content. If the page key is not found, this will 404.
Restricted access: YES. Search for Files permission needed. Additional search permission limits may apply.
@@ -999,8 +1000,8 @@
Restricted access: YES. Search for Files permission needed. Additional search permission limits may apply.
@@ -1158,8 +1159,8 @@
Restricted access: YES. Search for Files permission needed. Additional search permission limits may apply.
@@ -1182,8 +1183,8 @@
Response description: The file itself. You should get the correct mime type as the Content-Type header.
Restricted access: YES. Search for Files permission needed. Additional search permission limits may apply.
diff --git a/help/database_migration.html b/help/database_migration.html
index ddd95dda..8659fbec 100644
--- a/help/database_migration.html
+++ b/help/database_migration.html
@@ -6,7 +6,7 @@
-
@@ -25,10 +25,10 @@
Thumbnails tend to be fetched dozens at a time, so it is, again, ideal if they are stored on an SSD. Your regular media files--which on many clients total hundreds of GB--are usually fetched one at a time for human consumption and do not benefit from the expensive low-latency of an SSD. They are best stored on a cheap HDD, and, if desired, also work well across a network file system.
- Move your 'fast' files to the fast location. @@ -83,7 +83,7 @@
-
+
There is now a simple 'session' system, where you can get a temporary key that gives the same access without having to include the permanent access key in every request. You can fetch a session key with the /session_key command and thereafter use it just as you would an access key, just with Hydrus-Client-API-Session-Key instead.
Session keys will expire if they are not used within 24 hours, or if the client is restarted, or if the underlying access key is deleted. An invalid/expired session key will give a 419 result with an appropriate error text.
Bear in mind the Client API is still under construction and is http-only for the moment--be careful about transmitting sensitive content outside of localhost. The access key will be unencrypted across any connection, and if it is included as a GET parameter, as simple and convenient as that is, it could be cached in all sorts of places.
->Contents
+Contents
Access Management
-GET /api_version
+Access Management
+GET /api_version
Gets the current API version. I will increment this every time I alter the API.
GET /request_new_permissions
+GET /request_new_permissions
Register a new external program with the client. This requires the 'add from api request' mini-dialog under services->review services to be open, otherwise it will 403.
GET /session_key
+GET /session_key
Get a new session key.
GET /verify_access_key
+GET /verify_access_key
Check your access key is valid.
Adding Files
-POST /add_files/add_file
+Adding Files
+POST /add_files/add_file
Tell the client to import a file.
POST /add_files/delete_files
+POST /add_files/delete_files
Tell the client to send files to the trash.
POST /add_files/undelete_files
+POST /add_files/undelete_files
Tell the client to pull files back out of the trash.
POST /add_files/archive_files
+POST /add_files/archive_files
Tell the client to archive inboxed files.
POST /add_files/unarchive_files
+POST /add_files/unarchive_files
Tell the client re-inbox archived files.
Adding Tags
-GET /add_tags/clean_tags
+Adding Tags
+GET /add_tags/clean_tags
Ask the client about how it will see certain tags.
GET /add_tags/get_tag_services
+GET /add_tags/get_tag_services
Ask the client about its tag services.
POST /add_tags/add_tags
+POST /add_tags/add_tags
Make changes to the tags that files have.
Note also that hydrus tag actions are safely idempotent. You can pend a tag that is already pended and not worry about an error--it will be discarded. The same for other reasonable logical scenarios: deleting a tag that does not exist will silently make no change, pending a tag that is already 'current' will again be passed over. It is fine to just throw 'process this' tags at every file import you add and not have to worry about checking which files you already added it to.
Adding URLs
-GET /add_urls/get_url_files
+Adding URLs
+GET /add_urls/get_url_files
Ask the client about an URL's files.
GET /add_urls/get_url_info
+GET /add_urls/get_url_info
Ask the client for information about a URL.
POST /add_urls/add_url
+POST /add_urls/add_url
Tell the client to 'import' a URL. This triggers the exact same routine as drag-and-dropping a text URL onto the main client window.
POST /add_urls/associate_url
+POST /add_urls/associate_url
Manage which URLs the client considers to be associated with which files.
Managing Cookies
+Managing Cookies
This refers to the cookies held in the client's session manager, which are sent with network requests to different domains.
-GET /manage_cookies/get_cookies
+GET /manage_cookies/get_cookies
Get the cookies for a particular domain.
This request will also return any cookies for subdomains. The session system in hydrus generally stores cookies according to the second-level domain, so if you request for specific.someoverbooru.net, you will still get the cookies for someoverbooru.net and all its subdomains.
POST /manage_cookies/set_cookies
+POST /manage_cookies/set_cookies
Set some new cookies for the client. This makes it easier to 'copy' a login from a web browser or similar to hydrus if hydrus's login system can't handle the site yet.
Expires can be null, but session cookies will time-out in hydrus after 60 minutes of non-use.
Managing Pages
+Managing Pages
This refers to the pages of the main client UI.
-GET /manage_pages/get_pages
+GET /manage_pages/get_pages
Get the page structure of the current UI session.
The 'page_key' is a unique identifier for the page. It will stay the same for a particular page throughout the session, but new ones are generated on a client restart or other session reload.
GET /manage_pages/get_page_info
+GET /manage_pages/get_page_info
Get information about a specific page.
This is under construction. The current call dumps a ton of info for different downloader pages. Please experiment in IRL situations and give feedback for now! I will flesh out this help with more enumeration info and examples as this gets nailed down. POST commands to alter pages (adding, removing, highlighting), will come later.
-
@@ -929,8 +930,8 @@
POST /manage_pages/focus_page
+POST /manage_pages/focus_page
'Show' a page in the main GUI, making it the current page in view. If it is already the current page, no change is made.
Searching Files
+Searching Files
File search in hydrus is not paginated like a booru--all searches return all results in one go. In order to keep this fast, search is split into two steps--fetching file identifiers with a search, and then fetching file metadata in batches. You may have noticed that the client itself performs searches like this--thinking a bit about a search and then bundling results in batches of 256 files before eventually throwing all the thumbnails on screen.
-GET /get_files/search_files
+GET /get_files/search_files
Search for the client's files.
Note that most clients will have an invisible system:limit of 10,000 files on all queries. I expect to add more system predicates to help searching for untagged files, but it is tricky to fetch all files under any circumstance. Large queries may take several seconds to respond.
GET /get_files/file_metadata
+GET /get_files/file_metadata
Get metadata about files in the client.
GET /get_files/file
+GET /get_files/file
Get a file.
GET /get_files/thumbnail
+GET /get_files/thumbnail
Get a file's thumbnail.
the hydrus database
+the hydrus database
A hydrus client consists of three components:
these components can be put on different drives
+these components can be put on different drives
Although an initial install will keep these parts together, it is possible to, say, run the database on a fast drive but keep your media in cheap slow storage. This is an excellent arrangement that works for many users. And if you have a very large collection, you can even spread your files across multiple drives. It is not very technically difficult, but I do not recommend it for new users.
Backing such an arrangement up is obviously more complicated, and the internal client backup is not sophisticated enough to capture everything, so I recommend you figure out a broader solution with a third-party backup program like FreeFileSync.
-pulling your media apart
+pulling your media apart
As always, I recommend creating a backup before you try any of this, just in case it goes wrong.
If you would like to move your files and thumbnails to new locations, I generally recommend you not move their folders around yourself--the database has an internal knowledge of where it thinks its file and thumbnail folders are, and if you move them while it is closed, it will become confused and you will have to manually relocate what is missing on the next boot via a repair dialog. This is not impossible to figure out, but if the program's 'client files' folder confuses you at all, I'd recommend you stay away. Instead, you can simply do it through the gui:
Go database->migrate database, giving you this dialog:
@@ -39,7 +39,7 @@Weight means the relative amount of media you would like to store in that location. It only matters if you are spreading your files across multiple locations. If location A has a weight of 1 and B has a weight of 2, A will get approximately one third of your files and B will get approximately two thirds.
The operations on this dialog are simple and atomic--at no point is your db ever invalid. Once you have the locations and ideal usage set how you like, hit the 'move files now' button to actually shuffle your files around. It will take some time to finish, but you can pause and resume it later if the job is large or you want to undo or alter something.
If you decide to move your actual database, the program will have to shut down first. Before you boot up again, you will have to create a new program shortcut:
-informing the software that the database is not in the default location
+informing the software that the database is not in the default location
A straight call to the client executable will look for a database in install_dir/db. If one is not found, it will create one. So, if you move your database and then try to run the client again, it will try to create a new empty database in the previous location!
So, pass it a -d or --db_dir command line argument, like so:
-
@@ -53,9 +53,9 @@
Rather than typing the path out in a terminal every time you want to launch your external database, create a new shortcut with the argument in. Something like this, which is from my main development computer and tests that a fresh default install will run an existing database ok:
Note that an install with an 'external' database no longer needs access to write to its own path, so you can store it anywhere you like, including protected read-only locations (e.g. in 'Program Files'). If you do move it, just double-check your shortcuts are still good and you are done.
-finally
+finally
If your database now lives in one or more new locations, make sure to update your backup routine to follow them!
-moving to an SSD
+moving to an SSD
As an example, let's say you started using the hydrus client on your HDD, and now you have an SSD available and would like to move your thumbnails and main install to that SSD to speed up the client. Your database will be valid and functional at every stage of this, and it can all be undone. The basic steps are:
You should now have something like this:
-p.s. running multiple clients
+p.s. running multiple clients
Since you now know how to tell the software about an external database, you can, if you like, run multiple clients from the same install (and if you previously had multiple install folders, now you can now just use the one). Just make multiple shortcuts to the same client executable but with different database directories. They can run at the same time. You'll save yourself a little memory and update-hassle. I do this on my laptop client to run a regular client for my media and a separate 'admin' client to do PTR petitions and so on.
putting it all together
+putting it all together
Now you know what GUGs, URL Classes, and Parsers are, you should have some ideas of how URL Classes could steer what happens when the downloader is faced with an URL to process. Should a URL be imported as a media file, or should it be parsed? If so, how?
You may have noticed in the Edit GUG ui that it lists if a current URL Class matches the example URL output. If the GUG has no matching URL Class, it won't be listed in the main 'gallery selector' button's list--it'll be relegated to the 'non-functioning' page. Without a URL Class, the client doesn't know what to do with the output of that GUG. But if a URL Class does match, we can then hand the result over to a parser set at network->downloader definitions->manage url class links:
@@ -18,4 +18,4 @@