diff --git a/docs/advanced_sidecars.md b/docs/advanced_sidecars.md
new file mode 100644
index 00000000..89218f19
--- /dev/null
+++ b/docs/advanced_sidecars.md
@@ -0,0 +1,134 @@
+---
+title: Sidecars
+---
+
+# sidecars
+
+Sidecars are files that provide additional metadata about a master file. They typically share the same basic filename--if the master is 'Image_123456.jpg', the sidecar will be something like 'Image_123456.txt' or 'Image_123456.jpg.json'. This obviously makes it easy to figure out which sidecar goes with which file.
+
+Hydrus does not use sidecars in its own storage, but it can import data from them and export data to them. It currently supports raw data in .txt files and encoded data in .json files, and that data can be either tags or URLs. I expect to extend this system in future to support XML and other metadata types such as ratings, timestamps, and inbox/archive status.
+
+We'll start with .txt, since they are simpler.
+
+## Importing Sidecars
+
+Imagine you have some jpegs you downloaded with another program. That program grabbed the files' tags somehow, and you want to import the files with their tags without messing around with the Client API.
+
+If your extra program can export the tags to a simple format--let's say newline-separated .txt files with the same basic filename as the jpegs, or you can, with some very simple scripting, convert to that format--then importing them to hydrus is easy!
+
+Put the jpegs and the .txt files in the same directory and then drag and drop the directory onto the client, as you would for a normal import. The .txt files should not be added to the list. Then click 'add tags/urls with the import'. The sidecars are managed on one of the tabs:
+
+[![](images/sidecars_example_manual_import.png)](images/sidecars_example_manual_import.png)
+
+This system can get quite complicated, but the essential idea is that you are selecting one or more sidecar `sources`, parsing their text, and sending that list of data to one hydrus service `destination`. Most of the time you will be pulling from just one sidecar at a time.
+
+### The Source Dialog
+
+The `source` is a description of a sidecar to load and how to read what it contains.
+
+In this example, the texts are like so:
+
+``` title="4e01850417d1978e6328d4f40c3b550ef582f8558539b4ad46a1cb7650a2e10b.jpg.txt"
+flowers
+landscape
+blue sky
+```
+
+``` title="5e390f043321de57cb40fd7ca7cf0cfca29831670bd4ad71622226bc0a057876.jpg.txt"
+fast car
+anime girl
+night sky
+```
+
+Since our sidecars in this example are named (filename.ext).txt, and use newlines as the separator character, we can leave things mostly as default.
+
+If you do not have newline-separated tags, for instance comma-separated tags (`flowers, landscape, blue sky`), then you can set that here. Be careful if you are making your own sidecars, since any separator character obviously cannot be used in tag text!
+
+If your sidecars are named (filename).txt instead of (filename.ext).txt, then just hit the checkbox, but if the conversion is more complicated, then play around with the filename string converter and the test boxes.
+
+If you need to, you can further process the texts that are loaded. They'll be trimmed of extra whitespace and so on automatically, so no need to worry about that, but if you need to, let's say, add the `creator:` prefix to everything, or filter out some mis-parsed garbage, this is the place.
+
+### The Router Dialog
+
+A 'Router' is a single set of orders to grab from one or more sidecars and send to a destination. You can have several routers in a single import or export context.
+
+You can do more string processing here, and it will apply to everything loaded from every sidecar.
+
+The destination is either a tag service (adding the loaded strings as tags), or your known URLs store.
+
+### Previewing
+
+Once you have something set up, you can see the results are live-loaded in the dialog. Make sure everything looks all correct, and then start the import as normal and you should see the tags or URLs being added as the import works.
+
+It is good to try out some simple situations with one or two files just to get a feel for the system.
+
+### Import Folders
+
+If you have a constant flow of sidecar-attached media, then you can add sidecars to Import Folders too. Do a trial-run of anything you want to parse with a manual import before setting up the automatic system.
+
+## Exporting Sidecars
+
+The rules for exporting are similar, but now you are pulling from one or more hydrus service `sources` and sending to a single `destination` sidecar every time. Let's look at the UI:
+
+[![](images/sidecars_example_manual_export.png)](images/sidecars_example_manual_export.png)
+
+I have chosen to select these files' URLs and send them to newline-separated .urls.txt files. If I wanted to get the tags too, I could pull from one or more tag services, filter and convert the tags as needed, and then output to a .tags.txt file.
+
+The best way to learn with this is just to experiment. The UI may seem intimidating, but most jobs don't need you to work with multiple sidecars or string processing or clever filenames.
+
+## JSON Files
+
+JSON is more complicated than .txt. You might have multiple metadata types all together in one file, so you may end up setting up multiple routers that parse the same file for different content, or for an export you might want to populate the same export file with multiple kinds of content. Hydrus can do it!
+
+### Importing
+
+Since JSON files are richly structured, we will have to dip into the Hydrus parsing system:
+
+[![](images/sidecars_example_json_import.png)](images/sidecars_example_json_import.png)
+
+If you have made a downloader before, you will be familiar with this. If not, then you can brave [the help](downloader_parsers_formulae.md#json_formula) or just have a play around with the UI. In this example, I am getting the URL(s) of each JSON file, which are stored in a list under the `file_info_urls` key.
+
+It is important to paste an example JSON file that you want to parse into the parsing testing area (click the paste button) so you can test on read data live.
+
+Once you have the parsing set up, the rest of the sidecar UI is the same as for .txt. The JSON Parsing formula is just the replacement/equivalent for the .txt 'separator' setting.
+
+_Note that you could set up a second Router to import the tags from this file!_
+
+### Exporting
+
+In Hydrus, the exported JSON is typically a nested Object with a similar format as in the Import example. You set the names of the Object keys.
+
+[![](images/sidecars_example_json_export.png)](images/sidecars_example_json_export.png)
+
+Here I have set the URLs of each file to be stored under `metadata->urls`, which will make this sort of structure:
+
+``` json
+{
+ "metadata" : {
+ "urls" : [
+ "http://example.com/123456",
+ "https://site.org/post/45678"
+ ]
+ }
+}
+```
+
+The cool thing about JSON files is I can export multiple times to the same file and it will update it! Lets say I made a second Router that grabbed the tags, and it was set to export to the same filename but under `metadata->tags`. The final sidecar would look like this:
+
+``` json
+{
+ "metadata" : {
+ "tags" : [
+ "blonde hair",
+ "blue eyes",
+ "skirt"
+ ],
+ "urls" : [
+ "http://example.com/123456",
+ "https://site.org/post/45678"
+ ]
+ }
+}
+```
+
+You should be careful that the location you are exporting to does not have any old JSON files with conflicting filenames in it--hydrus will update them, not overwrite them! This may be an issue if you have an synchronising Export Folder that exports random files with the same filenames.
diff --git a/docs/changelog.md b/docs/changelog.md
index fa310d3b..65ac0820 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -7,6 +7,95 @@ title: Changelog
!!! note
This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
+## [Version 514](https://github.com/hydrusnetwork/hydrus/releases/tag/v514)
+
+### downloaders
+
+* twitter took down the API we were using, breaking all our nice twitter downloaders! argh!
+* a user has figured out a basic new downloader that grabs the tweets amongst the first twenty tweets-and-retweets of an account. yes, only the first twenty max, and usually fewer. because this is a big change, the client will ask about it when you update. if you have some complicated situation where you are working on the old default twitter downloaders and don't want them deleted, you can select 'no' on the dialog it throws up, but everyone else wants to say 'yes'. then check your twitter subs: make sure they moved to the new downloader, and you probably want to make them check more frequently too.
+* given the rate of changes at twitter, I think we can expect more changes and blocks in future. I don't know whether nitter will be viable alternative, so if the artists you like end up on a nice simple booru _anywhere_, I strongly recommend just moving there. twitter appears to be explicitly moving to non-third-party-friendly
+* thanks to a user's work, the 'danbooru - get webm ugoira' parser is fixed!
+* thanks to a user's work, the deviant art parser is updated to get the highest res image in more situations!
+* thanks to a user's work, the pixiv downloader now gets the artist note, in japanese (and translated, if there is one), and a 'medium:ai generated' tag!
+
+### sidecars
+
+* I wrote some sidecar help here! https://hydrusnetwork.github.io/hydrus/advanced_sidecars.html
+* when the client parses files for import, the 'does this look like a sidecar?' test now also checks that the base component of the base filename (e.g. 'Image123' from 'Image123.jpg.txt') actually appears in the list of non-txt/json/xml ext files. a random yo.txt file out of nowhere will now be inspected in case it is secretly a jpeg again, for good or ill
+* when you drop some files on the client, the number of files skipped because they looked like sidecars is now stated in the status label
+* fixed a typo bug that meant tags imported from sidecars were not being properly cleaned, despite preview appearance otherwise, for instance ':)', which in hydrus needs to be secretly stored as '::)' was being imported as ')'
+* as a special case, tags that in hydrus are secretly '::)' will be converted to ':)' on export to sidecar too, the inverse of the above problem. there may be some other tag cleaning quirks to undo here, so let me know what you run into
+
+### related tags overhaul
+
+* the 'related tags' suggestion system, turned on under _options->tag suggestions_, has several changes, including some prototype tech I'd love feedback on
+* first off, there are two new search buttons, 'new 1' and 'new 2' ('2' is available on repositories only).. these use an upgraded statistical search and scoring system that a user worked on and sent in. I have butchered his specific namespace searching system to something more general/flexible and easy for me to maintain, but it works better and more comprehensibly than my old method! give it a go and let me know how each button does--the first one will be fast but less useful on the PTR, the second will be slower but generally give richer results (although it cannot do tags with too-high count)
+* the new search routine works on multiple files, so 'related tags' now shows on tag dialogs launched from a selection of thumbnails!
+* also, all the related search buttons now search any selection of tags you make!!! so if you can't remember that character's name, just click on the series or another character they are often with and hit the search, and you should get a whole bunch appear
+* I am going to keep working on this in the future. the new buttons will become the only buttons, I'll try and mitigate the prototype search limitations, add some cancel tech, move to a time-based search length like the current buttons, and I'll add more settings, including for filtering so we aren't looking up related tags for 'page:x' and so on. I'm interested in knowing how you get on with IRL data. are there too many recommendations (is the tolerance too high?)? is the sorting good (is the stuff at the top relevant or often just noise?)?
+
+### misc
+
+* all users can now copy their service keys (which are a technical non-changing hex identifier for your client's services) from the review services window--advanced mode is no longer needed. this may be useful as the client api transitions to service keys
+* when a job in the downloader search log generates new jobs (e.g. fetches the next page), the new job(s) are now inserted after the parent. previously, they were appended to the end of the list. this changes how ngugs operate, converting their searches from interleaved to sequential!
+* restarting search log jobs now also places the new job after the restarted job
+* when you create a new export folder, if you have default metadata export sidecar settings from a previous manual file export, the program now asks if you want those for the new export folder or an empty list. previously, it just assigned the saved default, which could be jarring if it was saved from ages ago
+* added a migration guide to the running from source help. also brushed up some language and fixed a bunch of borked title weights in that document
+* the max initial and periodic file limits in subscriptions is now 50k when in advanced mode. I can't promise that would be nice though!
+* the file history chart no longer says that inbox and delete time tracking are new
+
+### misc fixes
+
+* fixed a cursor type detection test that was stopping the cursor from hiding immediately when you do a media viewer drag in Qt6
+* fixed an issue where 'clear deletion record' calls were not deleting from the newer 'all my files' domain. the erroneous extra records will be searched for and scrubbed on update
+* fixed the issue where if you had the new 'unnamespaced input gives (any namespace) wildcard results' search option on, you couldn't add any novel tags in WRITE autocomplete contexts like 'manage tags'!!! it could only offer the automatically converted wildcard tags as suggested input, which of course aren't appropriate for a WRITE context. the way I ultimately fixed this was horrible; the whole thing needs more work to deal with clever logic like this better, so let me know if you get any more trouble here
+* I think I fixed an infinite hang when trying to add certain siblings in manage tag siblings. I believe this was occuring when the dialog was testing if the new pair would create a loop when the sibling structure already contains a loop. now it throws up a message and breaks the test
+* fixed an issue where certain system:filetype predicates would spawn apparent duplicates of themselves instead of removing on double-click. images+audio+video+swf+pdf was one example. it was a 'all the image types' vs 'list of (all the) image types' conversion/comparison/sorting issue
+
+### client api
+
+* **this is later than I expected, but as was planned last year, I am clearing up several obsolete parameters and data structures this week. mostly it is bad service name-identification that seemed simple or flexible to support but just added maintenance debt, induced bad implementation practises, and hindered future expansions. if you have a custom api script, please read on--and if you have not yet moved to the alternatives, do so before updating!**
+* **all `...service_name...` parameters are officially obsolete! they will still work via some legacy hacks, so old scripts shouldn't break, but they are no longer documented. please move to the `...service_key...` alternates as soon as reasonably possible (check out `/get_services` if you need to learn about service keys)**
+* **`/add_tags/get_tag_services` is removed! use `/get_services` instead!**
+* **`hide_service_names_tags`, previously made default true, is removed and its data structures `service_names_to_statuses_to_...` are also gone! move to the new `tags` structure.**
+* **`hide_service_keys_tags` is now default true. it will be removed in 4 weeks or so. same deal as with `service_names_to_statuses_to_...`--move to `tags`**
+* **`system_inbox` and `system_archive` are removed from `/get_files/search_files`! just use 'system:inbox/archive' in the tags list**
+* **the 'set_file_relationships' command from last week has been reworked to have a nicer Object parameter with a new name. please check the updated help!** normally I wouldn't change something so quick, but we are still in early prototype, so I'm ok shifting it (and the old method still works lmao, but I'll clear that code out in a few weeks, so please move over--the Object will be much nicer to expand in future, which I forgot about in v513)
+### many Client API commands now support modern file domain objects, meaning you can search a UNION of file services and 'deleted-from' file services. affected commands are
+
+* * /add_files/delete_files
+* * /add_files/undelete_files
+* * /add_tags/search_tags
+* * /get_files/search_files
+* * /manage_file_relationships/get_everything
+* a new `/get_service` call now lets you ask about an individual service by service name or service key, basically a parameterised /get_services
+* the `/manage_pages/get_pages` and `/manage_pages/get_page_info` calls now give the `page_state`, a new enum that says if the page is ready, initialised, searching, or search-cancelled
+* to reduce duplicate argument spam, the client api help now specifies the complicated 'these files' and now 'this file domain' arguments into sub-sections, and the commands that use them just point to the subsections. check it out--it makes sense when you look at it.
+* `/add_tags/add_tags` now raises 400 if you give an invalid content action (e.g. pending to a local tag service). previously it skipped these rows silently
+* added and updated unit tests and help for the above changes
+* client api version is now 41
+
+### boring optimisation
+
+* when you are looking at a search log or file log, if entries are added, removed, or moved around, all the log entries that have changed row # now update (previously it just sent a redraw signal for the new rows, not the second-order affected rows that were shuffled up/down. many access routines for these logs are sped up
+* file log status checking is completely rewritten. the ways it searches, caches and optimises the 'which is the next item with x status' queues is faster and requires far less maintenance. large import queues have less overhead, so the in and outs of general download work should scale up much better now
+* the main data cache that stores rendered images, image tiles, and thumbnails now maintains itself far more efficiently. there was a hellish O(n) overhead when adding or removing an item which has been reduced to constant time. this gonk was being spammed every few minutes during normal memory maintenance, when hundreds of thumbs can be purged at once. clients with tens of thousands of thumbnails in memory will maintain that list far more smoothly
+* physical file delete is now more efficient, requiring far fewer hard drive hits to delete a media file. it is also far less aggressive, with a new setting in _options->files and trash_ that sets how long to wait between individual file deletes, default 250ms. before, it was full LFG mode with minor delays every hundred/thousand jobs, and since it takes a write lock, it was lagging out thumbnail load when hitting a lot of work. the daemon here also shuts down faster if caught working during program shut down
+
+### boring code cleanup
+
+* refactored some parsing routines to be more flexible
+* added some more dictionary and enum type testing to the client api parameter parsing routines. error messages should be better!
+* improved how `/add_tags/add_tags` parsing works. ensuring both access methods check all types and report nicer errors
+* cleaned up the `/search_files/file_metadata` call's parsing, moving to the new generalised method and smoothing out some old code flow. it now checks hashes against the last search, too
+* cleaned up `/manage_pages/add_files` similarly
+* cleaned up how tag services are parsed and their errors reported in the client api
+* the client api is better about processing the file identifiers you give it in the same order you gave
+* fixed bad 'potentials_search_type'/'search_type' inconsistency in the client api help examples
+* obviously a bunch of client api unit test and help cleanup to account for the obsolete stuff and various other changes here
+* updated a bunch of the client api unit tests to handle some of the new parsing
+* fixed the remaining 'randomly fail due to complex counting logic' potential count unit tests. turns out there were like seven more of them
+
## [Version 513](https://github.com/hydrusnetwork/hydrus/releases/tag/v513)
### client api
@@ -430,52 +519,3 @@ title: Changelog
* cleaned up some edge cases in the 'which account added this file/mapping to the server?' tech, where it might have been possible, when looking up deleted content, to get another janitor account (i.e. who deleted the content), although I am pretty sure this situation was never possible to actually start in UI. if I add 'who deleted this?' tech in future, it'll be a separate specific call
* cleaned up some specifically 'Qt6' references in the build script. the build requirements.txts and spec files are also collapsed down, with old Qt5 versions removed
* filled out some incomplete abstract class definitions
-
-## [Version 504](https://github.com/hydrusnetwork/hydrus/releases/tag/v504)
-
-### Qt5
-* as a reminder, I am no longer supporting Qt5 with the official builds. if you are on Windows 7 (and I have heard at least one version of Win 8.1), or a similarly old OS, you likely cannot run the official builds now. if this is you, please check the 'running from source' guide in the help, which will allow you to keep updating the program. this process is now easy in Windows and should be similarly easy on other platforms soon
-
-### misc
-* if you run from source in windows, the program _should_ now have its own taskbar group and use the correct hydrus icon. if you try and pin it to taskbar, it will revert to the 'python' icon, but you can give a shortcut to a batch file an icon and pin that to start
-* unfortunately, I have to remove the 'deviant art tag search' downloader this week. they killed the old API we were using, and what remaining open date-paginated search results the site offers is obfuscated and tokenised (no permanent links), more than I could quickly unravel. other downloader creators are welcome to give it a go. if you have a subscription for a da tag search, it will likely complain on its next run. please pause it and try to capture the best artists from that search (until DA kill their free artist api, then who knows what will happen). the oauth/phone app menace marches on
-* focus on the thumbnail panel is now preserved whenever it swaps out for another (like when you refresh the search)
-* fixed an issue where cancelling service selection on database->c&r->repopulate truncated would create an empty modal message
-* fixed a stupid typo in the recently changed server petition counting auto-fixing code
-
-### importer/exporter sidecar expansion
-* when you import or export files from/to disk, either manually or automatically, the option to pull or send tags to .txt files is now expanded:
-* - you can now import or export URLs
-* - you can now read or write .json files
-* - you can now import from or export to multiple sidecars, and have multiple separate pipelines
-* - you can now give sidecar files suffixes, for ".tags.txt" and similar
-* - you can now filter and transform all the strings in this pipeline using the powerful String Processor just like in the parsing system
-* this affects manual imports, manual exports, import folders, and export folders. instead of smart .txt checkboxes, there's now a button leading to some nested dialogs to customise your 'routers' and, in manual imports, a new page tab in the 'add tags before import' window
-* this bones of this system was already working in the background when I introduced it earlier this year, but now all components are exposed
-* new export folders now start with the same default metadata migration as set in the last manual file export dialog
-* this system will expand in future. most important is to add a 'favourites' system so you can easily save/load your different setups. then adding more content types (e.g. ratings) and .xml. I'd also like to add purely internal file-to-itself datatype transformation (e.g. pulling url:(url) tags and converting them to actual known urls, and vice versa)
-
-### importer/exporter sidecar expansion (boring stuff)
-* split the importer/exporter objects into separate importers and exporters. existing router objects will update and split their internal objects safely
-* all objects in this system can now describe themselves
-* all import/export nodes now produce appropriate example texts for string processing and parsing UI test panels
-* Filename Tagging Options objects no longer track neighbouring .txt file importing, and their UI removes it too. Import Folders will suck their old data on update and convert to metadata routers
-* wrote a json sidecar importer that takes a parsing formula
-* wrote a json sidecar exporter that takes a list of dictionary names to export to. it will edit an existing file
-* wrote some ui panels to edit single file metadata migration routers
-* wrote some ui panels to edit single file metadata migration importers
-* wrote some ui panels to edit single file metadata migration exporters
-* updated edit export folder panel to use the new UI. it was already using a full static version of the system behind the scenes; now this is exposed and editable
-* updated the manual file export panel to use the new UI. it was using a half version of the system before--now the default options are updated to the new router object and you can create multiple exports
-* updated import folders to use the new UI. the filename tagging options no longer handles .txt, it is now on a separate button on the import folder
-* updated manual file imports to use the new UI. the 'add tags before import' window now has a 'sidecars' page tab, which lets you edit metadata routers. it updates a path preview list live with what it expects to parse
-* a full suite of new unit tests now checks the router, the four import nodes, and the four export nodes thoroughly
-* renamed ClientExportingMetadata to ClientMetadataMigration and moved to the metadata module. refactored the importers, exporters, and shared methods to their own files in the same module
-* created a gui.metadata module for the new router and metadata import/export widgets and panels
-* created a gui.exporting module for the existing export folder and manual export gui code
-* reworked some of the core importer/exporter objects and inheritance in clientmetadatamigration
-* updated the HDDImport object and creation pipeline to handle metadata routers (as piped from the new sidecars tab)
-* when the hdd import or import folder is set to delete original files, now all defined sidecars are deleted along with the media file
-* cleaned up a bunch of related metadata importer/exporter code
-* cleaned import folder code
-* cleaned hdd importer code
diff --git a/docs/developer_api.md b/docs/developer_api.md
index c7568003..c4df886e 100644
--- a/docs/developer_api.md
+++ b/docs/developer_api.md
@@ -106,6 +106,68 @@ Session keys will expire if they are not used within 24 hours, or if the client
Bear in mind the Client API is still under construction. Setting up the Client API to be accessible across the internet requires technical experience to be convenient. HTTPS is available for encrypted comms, but the default certificate is self-signed (which basically means an eavesdropper can't see through it, but your ISP/government could if they decided to target you). If you have your own domain and SSL cert, you can replace them though (check the db directory for client.crt and client.key). Otherwise, be careful about transmitting sensitive content outside of your localhost/network.
+## Common Complex Parameters
+
+### **files** { id="parameters_files" }
+
+If you need to refer to some files, you can use any of the following:
+
+Arguments:
+:
+ * `file_id`: (selective, a numerical file id)
+ * `file_ids`: (selective, a list of numerical file ids)
+ * `hash`: (selective, a hexadecimal SHA256 hash)
+ * `hashes`: (selective, a list of hexadecimal SHA256 hashes)
+
+In GET requests, make sure any list is percent-encoded.
+
+### **file domain** { id="parameters_file_domain" }
+
+When you are searching, you may want to specify a particular file domain. Most of the time, you'll want to just set `file_service_key`, but this can get complex:
+
+Arguments:
+:
+ * `file_service_key`: (optional, selective A, hexadecimal, the file domain on which to search)
+ * `file_service_keys`: (optional, selective A, list of hexadecimals, the union of file domains on which to search)
+ * `deleted_file_service_key`: (optional, selective B, hexadecimal, the 'deleted from this file domain' on which to search)
+ * `deleted_file_service_keys`: (optional, selective B, list of hexadecimals, the union of 'deleted from this file domain' on which to search)
+
+The service keys are as in [/get\_services](#get_services).
+
+Hydrus supports two concepts here:
+
+* Searching over a UNION of subdomains. If the user has several local file domains, e.g. 'favourites', 'personal', 'sfw', and 'nsfw', they might like to search two of them at once.
+* Searching deleted files of subdomains. You can specifically, and quickly, search the files that have been deleted from somewhere.
+
+You can play around with this yourself by clicking 'multiple locations' in the client with _help->advanced mode_ on.
+
+In extreme edge cases, these two can be mixed by populating both A and B selective, making a larger union of both current and deleted file records.
+
+Please note that unions can be very very computationally expensive. If you can achieve what you want with a single file_service_key, two queries in a row with different service keys, or an umbrella like `all my files` or `all local files`, please do. Otherwise, let me know what is running slow and I'll have a look at it.
+
+'deleted from all local files' includes all files that have been physically deleted (i.e. deleted from the trash) and not available any more for fetch file/thumbnail requests. 'deleted from all my files' includes all of those physically deleted files _and_ the trash. If a file is deleted with the special 'do not leave a deletion record' command, then it won't show up in a 'deleted from file domain' search!
+
+'all known files' is a tricky domain. It converts much of the search tech to ignore where files actually are and look at the accompanying tag domain (e.g. all the files that have been tagged), and can sometimes be very expensive.
+
+Also, if you have the option to set both file and tag domains, you cannot enter 'all known files'/'all known tags'. It is too complicated to support, sorry!
+
+### **legacy service_name parameters** { id="legacy_service_name_parameters" }
+
+The Client API used to respond to name-based service identifiers, for instance using 'my tags' instead of something like '6c6f63616c2074616773'. Service names can change, and they aren't _strictly_ unique either, so I have moved away from them, but there is some soft legacy support.
+
+The client will attempt to convert any of these to their 'service_key(s)' equivalents:
+
+* file_service_name
+* tag_service_name
+* service_names_to_tags
+* service_names_to_actions_to_tags
+* service_names_to_additional_tags
+
+But I strongly encourage you to move away from them as soon as reasonably possible. Look up the service keys you need with [/get\_service](#get_service) or [/get\_services](#get_services).
+
+If you have a clever script/program that does many things, then hit up [/get\_services](#get_services) on session initialisation and cache an internal map of key_to_name for the labels to use when you present services to the user.
+
+Also, note that all users can now copy their service keys from _review services_.
## Access Management
@@ -209,6 +271,44 @@ Response:
```
+### **GET `/get_service`** { id="get_service" }
+
+_Ask the client about a specific service._
+
+Restricted access:
+: YES. At least one of Add Files, Add Tags, Manage Pages, or Search Files permission needed.
+
+Required Headers: n/a
+
+Arguments:
+:
+ * `service_name`: (selective, string, the name of the service)
+ * `service_key`: (selective, hex string, the service key of the service)
+
+Example requests:
+:
+ ```title="Example requests"
+ /get_service?service_name=my%20tags
+ /get_service?service_key=6c6f63616c2074616773
+ ```
+
+Response:
+: Some JSON about the service. The same basic format as [/get\_services](#get_services)
+```json title="Example response"
+{
+ "service" : {
+ "name" : "my tags",
+ "service_key" : "6c6f63616c2074616773",
+ "type" : 5,
+ "type_pretty" : "local tag service"
+ }
+}
+```
+
+If the service does not exist, this gives 404. It is very unlikely but edge-case possible that two services will have the same name, in this case you'll get the pseudorandom first.
+
+It will only respond to services in the /get_services list. I will expand the available types in future as we add ratings etc... to the Client API.
+
### **GET `/get_services`** { id="get_services" }
_Ask the client about its file and tag services._
@@ -312,11 +412,14 @@ Response:
]
}
```
- These services may be referred to in various metadata responses or required in request parameters (e.g. where to add tag mappings). Note that a user can rename their services. The older parts of the Client API use the renameable 'service name' as service identifier, but wish to move away from this. Please use the hex 'service_key', which is a non-mutable ID specific to each client. The hardcoded services have shorter service key strings (it is usually just 'all known files' etc.. ASCII-converted to hex), but user-made stuff will have 64-character hex.
+
+ Note that a user can rename their services, so while they will recognise `name`, it is not an excellent identifier, and definitely not something to save to any permanent config file.
- Now that I state `type` and `type_pretty` here, I may rearrange this call, probably to make the `service_key` the Object key, rather than the arbitrary 'all_known_tags' strings.
+ `service_key` is non-mutable and is the main service identifier. The hardcoded/initial services have shorter fixed service key strings (it is usually just 'all known files' etc.. ASCII-converted to hex), but user-created services will have random 64-character hex.
- You won't see all these, and you'll only ever need some, but `type` is:
+ Now that I state `type` and `type_pretty` here, I may rearrange this call, probably into a flat list. The `all_known_files` Object keys here are arbitrary.
+
+ For service `type`, you won't see all these, and you'll only ever need some, but the enum is:
* 0 - tag repository
* 1 - file repository
@@ -337,6 +440,7 @@ Response:
* 21 - all my files -- union of all local file domains
* 99 - server administration
+ `type_pretty` is something you can show users if you like. Hydrus uses the same labels in _manage services_ and so on.
## Importing and Deleting Files
@@ -396,12 +500,8 @@ Required Headers:
Arguments (in JSON):
:
-* `hash`: (an SHA256 hash for a file in 64 characters of hexadecimal)
-* `hashes`: (a list of SHA256 hashes)
-* `file_id`: (a numerical file id)
-* `file_ids`: (a list of numerical file ids)
-* `file_service_name`: (optional, selective, string, the local file domain from which to delete, or all local files)
-* `file_service_key`: (optional, selective, hexadecimal, the local file domain from which to delete, or all local files)
+* [files](#parameters_files)
+* [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
* `reason`: (optional, string, the reason attached to the delete action)
```json title="Example request body"
@@ -411,9 +511,7 @@ Arguments (in JSON):
Response:
: 200 and no content.
-You can use hash or hashes, whichever is more convenient.
-
-If you specify a file service, the file will only be deleted from that location. Only local file domains are allowed (so you can't delete from a file repository or unpin from ipfs yet), but if you specific 'all local files', you should be able to trigger a physical delete if you wish.
+If you specify a file service, the file will only be deleted from that location. Only local file domains are allowed (so you can't delete from a file repository or unpin from ipfs yet). It defaults to 'all my files', which will delete from all local services (i.e. force sending to trash). Sending 'all local files' on a file already in the trash will trigger a physical file delete.
### **POST `/add_files/undelete_files`** { id="add_files_undelete_files" }
@@ -428,12 +526,8 @@ Required Headers:
Arguments (in JSON):
:
-* `hash`: (an SHA256 hash for a file in 64 characters of hexadecimal)
-* `hashes`: (a list of SHA256 hashes)
-* `file_id`: (a numerical file id)
-* `file_ids`: (a list of numerical file ids)
-* `file_service_name`: (optional, selective, string, the local file domain to which to undelete)
-* `file_service_key`: (optional, selective, hexadecimal, the local file domain to which to undelete)
+* [files](#parameters_files)
+* [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
```json title="Example request body"
{"hash" : "78f92ba4a786225ee2a1236efa6b7dc81dd729faf4af99f96f3e20bad6d8b538"}
@@ -444,7 +538,7 @@ Response:
You can use hash or hashes, whichever is more convenient.
-This is the reverse of a delete_files--removing files from trash and putting them back where they came from. If you specify a file service, the files will only be undeleted to there (if they have a delete record, otherwise this is nullipotent). If you do not specify a file service, they will be undeleted to all local file services for which there are deletion records. There is no error if any files do not currently exist in 'trash'.
+This is the reverse of a delete_files--removing files from trash and putting them back where they came from. If you specify a file service, the files will only be undeleted to there (if they have a delete record, otherwise this is nullipotent). The default, 'all my files', undeletes to all local file services for which there are deletion records. There is no error if any of the files do not currently exist in 'trash'.
### **POST `/add_files/archive_files`** { id="add_files_archive_files" }
@@ -460,10 +554,7 @@ Required Headers:
Arguments (in JSON):
:
-* `hash`: (an SHA256 hash for a file in 64 characters of hexadecimal)
-* `hashes`: (a list of SHA256 hashes)
-* `file_id`: (a numerical file id)
-* `file_ids`: (a list of numerical file ids)
+* [files](#parameters_files)
```json title="Example request body"
{"hash" : "78f92ba4a786225ee2a1236efa6b7dc81dd729faf4af99f96f3e20bad6d8b538"}
@@ -472,8 +563,6 @@ Arguments (in JSON):
Response:
: 200 and no content.
-You can use hash or hashes, whichever is more convenient.
-
This puts files in the 'archive', taking them out of the inbox. It only has meaning for files currently in 'my files' or 'trash'. There is no error if any files do not currently exist or are already in the archive.
@@ -490,10 +579,7 @@ Required Headers:
Arguments (in JSON):
:
-* `hash`: (an SHA256 hash for a file in 64 characters of hexadecimal)
-* `hashes`: (a list of SHA256 hashes)
-* `file_id`: (a numerical file id)
-* `file_ids`: (a list of numerical file ids)
+* [files](#parameters_files)
```json title="Example request body"
{"hash" : "78f92ba4a786225ee2a1236efa6b7dc81dd729faf4af99f96f3e20bad6d8b538"}
@@ -502,8 +588,6 @@ Arguments (in JSON):
Response:
: 200 and no content.
-You can use hash or hashes, whichever is more convenient.
-
This puts files back in the inbox, taking them out of the archive. It only has meaning for files currently in 'my files' or 'trash'. There is no error if any files do not currently exist or are already in the inbox.
@@ -616,10 +700,8 @@ Arguments (in JSON):
* `destination_page_key`: (optional page identifier for the page to receive the url)
* `destination_page_name`: (optional page name to receive the url)
* `show_destination_page`: (optional, defaulting to false, controls whether the UI will change pages on add)
- * `service_names_to_additional_tags`: (optional, selective, tags to give to any files imported from this url)
* `service_keys_to_additional_tags`: (optional, selective, tags to give to any files imported from this url)
* `filterable_tags`: (optional tags to be filtered by any tag import options that applies to the URL)
- * _`service_names_to_tags`: (obsolete, legacy synonym for service\_names\_to\_additional_tags)_
If you specify a `destination_page_name` and an appropriate importer page already exists with that name, that page will be used. Otherwise, a new page with that name will be recreated (and used by subsequent calls with that name). Make sure it that page name is unique (e.g. '/b/ threads', not 'watcher') in your client, or it may not be found.
@@ -627,7 +709,7 @@ Alternately, `destination_page_key` defines exactly which page should be used. B
`show_destination_page` defaults to False to reduce flicker when adding many URLs to different pages quickly. If you turn it on, the client will behave like a URL drag and drop and select the final page the URL ends up on.
-`service_names_to_additional_tags` and `service_keys_to_additional_tags` use the same data structure as in /add\_tags/add\_tags--service ids to a list of tags to add. You will need 'add tags' permission or this will 403. These tags work exactly as 'additional' tags work in a _tag import options_. They are service specific, and always added unless some advanced tag import options checkbox (like 'only add tags to new files') is set.
+`service_keys_to_additional_tags` uses the same data structure as in /add\_tags/add\_tags--service keys to a list of tags to add. You will need 'add tags' permission or this will 403. These tags work exactly as 'additional' tags work in a _tag import options_. They are service specific, and always added unless some advanced tag import options checkbox (like 'only add tags to new files') is set.
filterable_tags works like the tags parsed by a hydrus downloader. It is just a list of strings. They have no inherant service and will be sent to a _tag import options_, if one exists, to decide which tag services get what. This parameter is useful if you are pulling all a URL's tags outside of hydrus and want to have them processed like any other downloader, rather than figuring out service names and namespace filtering on your end. Note that in order for a tag import options to kick in, I think you will have to have a Post URL URL Class hydrus-side set up for the URL so some tag import options (whether that is Class-specific or just the default) can be loaded at import time.
@@ -635,8 +717,8 @@ filterable_tags works like the tags parsed by a hydrus downloader. It is just a
{
"url" : "https://8ch.net/tv/res/1846574.html",
"destination_page_name" : "kino zone",
- "service_names_to_additional_tags" : {
- "my tags" : ["as seen on /tv/"]
+ "service_keys_to_additional_tags" : {
+ "6c6f63616c2074616773" : ["as seen on /tv/"]
}
}
```
@@ -703,16 +785,13 @@ Required Headers:
Arguments (in JSON):
:
- * `url_to_add`: (an url you want to associate with the file(s))
- * `urls_to_add`: (a list of urls you want to associate with the file(s))
- * `url_to_delete`: (an url you want to disassociate from the file(s))
- * `urls_to_delete`: (a list of urls you want to disassociate from the file(s))
- * `hash`: (an SHA256 hash for a file in 64 characters of hexadecimal)
- * `hashes`: (a list of SHA256 hashes)
- * `file_id`: (a numerical file id)
- * `file_ids`: (a list of numerical file ids)
+ * `url_to_add`: (optional, selective A, an url you want to associate with the file(s))
+ * `urls_to_add`: (optional, selective A, a list of urls you want to associate with the file(s))
+ * `url_to_delete`: (optional, selective B, an url you want to disassociate from the file(s))
+ * `urls_to_delete`: (optional, selective B, a list of urls you want to disassociate from the file(s))
+ * [files](#parameters_files)
- All of these are optional, but you obviously need to have at least one of `url` arguments and one of the `hash` arguments. The single/multiple arguments work the same--just use whatever is convenient for you. Unless you really know what you are doing with URL Classes, I strongly recommend you stick to associating URLs with just one single 'hash' at a time. Multiple hashes pointing to the same URL is unusual and frequently unhelpful.
+ The single/multiple arguments work the same--just use whatever is convenient for you. Unless you really know what you are doing with URL Classes, I strongly recommend you stick to associating URLs with just one single 'hash' at a time. Multiple hashes pointing to the same URL is unusual and frequently unhelpful.
```json title="Example request body"
{
"url_to_add" : "https://rule34.xxx/index.php?id=2588418&page=post&s=view",
@@ -756,32 +835,6 @@ Response:
Mostly, hydrus simply trims excess whitespace, but the other examples are rare issues you might run into. 'system' is an invalid namespace, tags cannot be prefixed with hyphens, and any tag starting with ':' is secretly dealt with internally as "\[no namespace\]:\[colon-prefixed-subtag\]". Again, you probably won't run into these, but if you see a mismatch somewhere and want to figure it out, or just want to sort some numbered tags, you might like to try this.
-### **GET `/add_tags/get_tag_services`** { id="add_tags_get_tag_services" }
-
-!!! warning "Deprecated"
- This is becoming obsolete and will be removed! Use [/get_services](#get_services) instead!
-
-_Ask the client about its tag services._
-
-Restricted access:
-: YES. Add Tags permission needed.
-
-Required Headers: n/a
-
-Arguments: n/a
-
-Response:
-: Some JSON listing the client's 'local tags' and tag repository services by name.
-```json title="Example response"
-{
- "local_tags" : ["my tags"],
- "tag_repositories" : [ "public tag repository", "mlp fanfic tagging server" ]
-}
-```
-
- !!! note
- A user can rename their services. Don't assume the client's local tags service will be "my tags".
-
### **GET `/add_tags/search_tags`** { id="add_tags_search_tags" }
_Search the client for tags._
@@ -793,15 +846,21 @@ Required Headers: n/a
Arguments:
:
-* `search`: (the tag text to search for, enter exactly what you would in the client UI)
-* `tag_service_key`: (optional, selective, hexadecimal, the tag domain on which to search)
-* `tag_service_name`: (optional, selective, string, the tag domain on which to search)
-* `tag_display_type`: (optional, string, to select whether to search raw or sibling-processed tags)
+ * `search`: (the tag text to search for, enter exactly what you would in the client UI)
+ * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
+ * `tag_service_key`: (optional, hexadecimal, the tag domain on which to search, defaults to 'all known tags')
+ * `tag_display_type`: (optional, string, to select whether to search raw or sibling-processed tags, defaults to 'storage')
+
+The `file domain` and `tag_service_key` perform the function of the file and tag domain buttons in the client UI.
+
+The `tag_display_type` can be either `storage` (the default), which searches your file's stored tags, just as they appear in a 'manage tags' dialog, or `display`, which searches the sibling-processed tags, just as they appear in a normal file search page. In the example above, setting the `tag_display_type` to `display` could well combine the two kim possible tags and give a count of 3 or 4.
+
+'all my files'/'all known tags' works fine for most cases, but a specific tag service or 'all known files'/'tag service' can work better for editing tag repository `storage` contexts, since it provides results just for that service, and for repositories, it gives tags for all the non-local files other users have tagged.
Example request:
:
```http title="Example request"
-/add_tags/search_tags?search=kim
+/add_tags/search_tags?search=kim&tag_display_type=display
```
Response:
@@ -827,14 +886,10 @@ Response:
}
```
-The `tags` list will be sorted by descending count. If you do not specify a tag service, it will default to 'all known tags'. The various rules in _tags->manage tag display and search_ (e.g. no pure `*` searches on certain services) will also be checked--and if violated, you will get 200 OK but an empty result.
-
-The `tag_display_type` can be either `storage` (the default), which searches your file's stored tags, just as they appear in a 'manage tags' dialog, or `display`, which searches the sibling-processed tags, just as they appear in a normal file search page. In the example above, setting the `tag_display_type` to `display` could well combine the two kim possible tags and give a count of 3 or 4.
+The `tags` list will be sorted by descending count. The various rules in _tags->manage tag display and search_ (e.g. no pure `*` searches on certain services) will also be checked--and if violated, you will get 200 OK but an empty result.
Note that if your client api access is only allowed to search certain tags, the results will be similarly filtered.
-Also, for now, it gives you the 'storage' tags, which are the 'raw' ones you see in the manage tags dialog, without collapsed siblings, but more options will be added in future.
-
### **POST `/add_tags/add_tags`** { id="add_tags_add_tags" }
_Make changes to the tags that files have._
@@ -846,21 +901,14 @@ Required Headers: n/a
Arguments (in JSON):
:
-* `hash`: (selective A, an SHA256 hash for a file in 64 characters of hexadecimal)
-* `hashes`: (selective A, a list of SHA256 hashes)
-* `file_id`: (a numerical file id)
-* `file_ids`: (a list of numerical file ids)
-* `service_names_to_tags`: (selective B, an Object of service names to lists of tags to be 'added' to the files)
+* [files](#parameters_files)
* `service_keys_to_tags`: (selective B, an Object of service keys to lists of tags to be 'added' to the files)
-* `service_names_to_actions_to_tags`: (selective B, an Object of service names to content update actions to lists of tags)
* `service_keys_to_actions_to_tags`: (selective B, an Object of service keys to content update actions to lists of tags)
- You can use either 'hash' or 'hashes'.
-
- You can use either 'service\_names\_to...' or 'service\_keys\_to...', where names is simple and human-friendly "my tags" and similar (but may be renamed by a user), but keys is a little more complicated but accurate/unique. Since a client may have multiple tag services with non-default names and pseudo-random keys, if it is not your client you will need to check the [/get_services](#get_services) call to get the names or keys, and you may need some selection UI on your end so the user can pick what to do if there are multiple choices. I encourage using keys if you can.
-
+ In 'service\_keys\_to...', the keys are as in [/get\_services](#get_services). You may need some selection UI on your end so the user can pick what to do if there are multiple choices.
+
Also, you can use either '...to\_tags', which is simple and add-only, or '...to\_actions\_to\_tags', which is more complicated and allows you to remove/petition or rescind pending content.
-
+
The permitted 'actions' are:
* 0 - Add to a local tag service.
@@ -877,8 +925,8 @@ Some example requests:
```json title="Adding some tags to a file"
{
"hash" : "df2a7b286d21329fc496e3aa8b8a08b67bb1747ca32749acb3f5d544cbfc0f56",
- "service_names_to_tags" : {
- "my tags" : ["character:supergirl", "rating:safe"]
+ "service_keys_to_tags" : {
+ "6c6f63616c2074616773" : ["character:supergirl", "rating:safe"]
}
}
```
@@ -888,9 +936,9 @@ Some example requests:
"df2a7b286d21329fc496e3aa8b8a08b67bb1747ca32749acb3f5d544cbfc0f56",
"f2b022214e711e9a11e2fcec71bfd524f10f0be40c250737a7861a5ddd3faebf"
],
- "service_names_to_tags" : {
- "my tags" : ["process this"],
- "public tag repository" : ["creator:dandon fuga"]
+ "service_keys_to_tags" : {
+ "6c6f63616c2074616773" : ["process this"],
+ "ccb0cf2f9e92c2eb5bd40986f72a339ef9497014a5fb8ce4cea6d6c9837877d9" : ["creator:dandon fuga"]
}
}
```
@@ -914,7 +962,7 @@ Some example requests:
This last example is far more complicated than you will usually see. Pend rescinds and petition rescinds are not common. Petitions are also quite rare, and gathering a good petition reason for each tag is often a pain.
- Note that the enumerated status keys in the service\_names\_to\_actions\_to_tags structure are strings, not ints (JSON does not support int keys for Objects).
+ Note that the enumerated status keys in the service\_keys\_to\_actions\_to_tags structure are strings, not ints (JSON does not support int keys for Objects).
Response description:
: 200 and no content.
@@ -1028,16 +1076,12 @@ Required Headers: n/a
Arguments (in percent-encoded JSON):
:
* `tags`: (a list of tags you wish to search for)
- * `file_service_name`: (optional, selective, string, the file domain on which to search)
- * `file_service_key`: (optional, selective, hexadecimal, the file domain on which to search)
- * `tag_service_name`: (optional, selective, string, the tag domain on which to search)
- * `tag_service_key`: (optional, selective, hexadecimal, the tag domain on which to search)
- * `file_sort_type`: (optional, integer, the results sort method)
+ * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
+ * `tag_service_key`: (optional, hexadecimal, the tag domain on which to search, defaults to 'all my files')
+ * `file_sort_type`: (optional, integer, the results sort method, defaults to 'all known tags')
* `file_sort_asc`: true or false (optional, the results sort order)
* `return_file_ids`: true or false (optional, default true, returns file id results)
* `return_hashes`: true or false (optional, default false, returns hex hash results)
- * _`system_inbox`: true or false (obsolete, use tags)_
- * _`system_archive`: true or false (obsolete, use tags)_
``` title='Example request for 16 files (system:limit=16) in the inbox with tags "blue eyes", "blonde hair", and "кино"'
/get_files/search_files?tags=%5B%22blue%20eyes%22%2C%20%22blonde%20hair%22%2C%20%22%5Cu043a%5Cu0438%5Cu043d%5Cu043e%22%2C%20%22system%3Ainbox%22%2C%20%22system%3Alimit%3D16%22%5D
@@ -1046,8 +1090,6 @@ Arguments (in percent-encoded JSON):
If the access key's permissions only permit search for certain tags, at least one positive whitelisted/non-blacklisted tag must be in the "tags" list or this will 403. Tags can be prepended with a hyphen to make a negated tag (e.g. "-green eyes"), but these will not be checked against the permissions whitelist.
-File searches occur in the `display` `tag_display_type`. If you want to pair autocomplete tag lookup from [/search_tags](#add_tags_search_tags) to this file search (e.g. for making a standard booru search interface), then make sure you are searching `display` tags there.
-
Wildcards and namespace searches are supported, so if you search for 'character:sam*' or 'series:*', this will be handled correctly clientside.
**Many system predicates are also supported using a text parser!** The parser was designed by a clever user for human input and allows for a certain amount of error (e.g. ~= instead of ≈, or "isn't" instead of "is not") or requires more information (e.g. the specific hashes for a hash lookup). **Here's a big list of examples that are supported:**
@@ -1109,8 +1151,10 @@ Wildcards and namespace searches are supported, so if you search for 'character:
* system:file service currently in my files
* system:file service is not currently in my files
* system:file service is not pending to my files
+ * system:number of file relationships = 2 duplicates
+ * system:number of file relationships > 10 potential duplicates
* system:num file relationships < 3 alternates
- * system:number of file relationships > 3 false positives
+ * system:num file relationships > 3 false positives
* system:ratio is wider than 16:9
* system:ratio is 16:9
* system:ratio taller than 1:1
@@ -1153,7 +1197,9 @@ Makes:
* samus aran OR lara croft
* system:height > 1000
-The file and tag services are for search domain selection, just like clicking the buttons in the client. They are optional--default is 'my files' and 'all known tags', and you can use either key or name as in [GET /get_services](#get_services), whichever is easiest for your situation.
+The file and tag services are for search domain selection, just like clicking the buttons in the client. They are optional--default is 'all my files' and 'all known tags'.
+
+File searches occur in the `display` `tag_display_type`. If you want to pair autocomplete tag lookup from [/search_tags](#add_tags_search_tags) to this file search (e.g. for making a standard booru search interface), then make sure you are searching `display` tags there.
file\_sort\_asc is 'true' for ascending, and 'false' for descending. The default is descending.
@@ -1242,24 +1288,20 @@ _Get metadata about files in the client._
Restricted access:
: YES. Search for Files permission needed. Additional search permission limits may apply.
-
+
Required Headers: n/a
-
+
Arguments (in percent-encoded JSON):
:
- * `file_id`: (selective, a numerical file id)
- * `file_ids`: (selective, a list of numerical file ids)
- * `hash`: (selective, a hexadecimal SHA256 hash)
- * `hashes`: (selective, a list of hexadecimal SHA256 hashes)
+ * [files](#parameters_files)
* `create_new_file_ids`: true or false (optional if asking with hash(es), defaulting to false)
* `only_return_identifiers`: true or false (optional, defaulting to false)
* `only_return_basic_information`: true or false (optional, defaulting to false)
* `detailed_url_information`: true or false (optional, defaulting to false)
* `include_notes`: true or false (optional, defaulting to false)
- * `hide_service_keys_tags`: **Will be set default false and deprecated soon!** true or false (optional, defaulting to false)
- * `hide_service_names_tags`: **Deprecated, will be deleted soon!** true or false (optional, defaulting to true)
+ * `hide_service_keys_tags`: **Deprecated, will be deleted soon!** true or false (optional, defaulting to true)
-You need one of file_ids or hashes. If your access key is restricted by tag, you cannot search by hashes, and **the file_ids you search for must have been in the most recent search result**.
+If your access key is restricted by tag, **the files you search for must have been in the most recent search result**.
``` title="Example request for two files with ids 123 and 4567"
/get_files/file_metadata?file_ids=%5B123%2C%204567%5D
@@ -1309,8 +1351,6 @@ Response:
"has_human_readable_embedded_metadata" : true,
"has_icc_profile" : true,
"known_urls" : [],
- "service_keys_to_statuses_to_tags" : {},
- "service_keys_to_statuses_to_display_tags" : {},
"tags" : {
"6c6f63616c2074616773" : {
"name" : "local tags",
@@ -1400,34 +1440,6 @@ Response:
"https://img2.gelbooru.com//images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg",
"http://origin-orig.deviantart.net/ed31/f/2019/210/7/8/beachqueen_samus_by_dandonfuga-ddcu1xg.jpg"
],
- "service_keys_to_statuses_to_tags" : {
- "6c6f63616c2074616773" : {
- "0" : ["samus favourites"],
- "2" : ["process this later"]
- },
- "37e3849bda234f53b0e9792a036d14d4f3a9a136d1cb939705dbcd5287941db4" : {
- "0" : ["blonde_hair", "blue_eyes", "looking_at_viewer"],
- "1" : ["bodysuit"]
- },
- "616c6c206b6e6f776e2074616773" : {
- "0" : ["samus favourites", "blonde_hair", "blue_eyes", "looking_at_viewer"],
- "1" : ["bodysuit"]
- }
- },
- "service_keys_to_statuses_to_display_tags" : {
- "6c6f63616c2074616773" : {
- "0" : ["samus favourites", "favourites"],
- "2" : ["process this later"]
- },
- "37e3849bda234f53b0e9792a036d14d4f3a9a136d1cb939705dbcd5287941db4" : {
- "0" : ["blonde hair", "blue_eyes", "looking at viewer"],
- "1" : ["bodysuit", "clothing"]
- },
- "616c6c206b6e6f776e2074616773" : {
- "0" : ["samus favourites", "favourites", "blonde hair", "blue_eyes", "looking at viewer"],
- "1" : ["bodysuit", "clothing"]
- }
- },
"tags" : {
"6c6f63616c2074616773" : {
"name" : "local tags",
@@ -1530,17 +1542,15 @@ Size is in bytes. Duration is in milliseconds, and may be an int or a float.
`ipfs_multihashes` stores the ipfs service key to any known multihash for the file.
-The `thumbnail_width` and `thumbnail_height` are a generally reliable prediction but aren't a promise. The actual thumbnail you get from [/get_files/thumbnail](#get_files_thumbnail) will be different if the user hasn't looked at it since changing their thumbnail options. You only get these rows for files that hydrus actually generates an actual thumbnail for. Things like pdf won't have it. You can use your own thumb, or ask the api and it'll give you a fixed fallback; those are mostly 200x200, but you can and should size them to whatever you want.
+The `thumbnail_width` and `thumbnail_height` are a generally reliable prediction but aren't a promise. The actual thumbnail you get from [/get\_files/thumbnail](#get_files_thumbnail) will be different if the user hasn't looked at it since changing their thumbnail options. You only get these rows for files that hydrus actually generates an actual thumbnail for. Things like pdf won't have it. You can use your own thumb, or ask the api and it'll give you a fixed fallback; those are mostly 200x200, but you can and should size them to whatever you want.
#### tags
The 'tags' structures are undergoing transition. Previously, this was a mess of different Objects in different domains, all `service_xxx_to_xxx_tags`, but they are being transitioned to the combined `tags` Object.
-`hide_service_names_tags` is deprecated and will be deleted soon. When set to `false`, it shows the old `service_names_to_statuses_to_tags` and `service_names_to_statuses_to_display_tags` Objects. The new `tags` structure now shows the service name--migrate to this asap.
+`hide_service_keys_tags` is deprecated and will be deleted soon. When set to `false`, it shows the old `service_keys_to_statuses_to_tags` and `service_keys_to_statuses_to_display_tags` Objects.
-`hide_service_keys_tags` will soon be set to default `false` and deprecated in the same way. Move to `tags` please!
-
-The `tags` structures are similar to the [/add_tags/add_tags](#add_tags_add_tags) scheme, excepting that the status numbers are:
+The `tags` structures are similar to the [/add\_tags/add\_tags](#add_tags_add_tags) scheme, excepting that the status numbers are:
* 0 - current
* 1 - pending
@@ -1550,7 +1560,7 @@ The `tags` structures are similar to the [/add_tags/add_tags](#add_tags_add_tags
!!! note
Since JSON Object keys must be strings, these status numbers are strings, not ints.
-To learn more about service names and keys on a client, use the [/get_services](#get_services) call.
+To learn more about service names and keys on a client, use the [/get\_services](#get_services) call.
While the 'storage_tags' represent the actual tags stored on the database for a file, 'display_tags' reflect how tags appear in the UI, after siblings are collapsed and parents are added. If you want to edit a file's tags, refer to the storage tags. If you want to render to the user, use the display tags. The display tag calculation logic is very complicated; if the storage tags change, do not try to guess the new display tags yourself--just ask the API again.
@@ -1665,7 +1675,7 @@ This refers to the File Relationships system, which includes 'potential duplicat
This system is pending significant rework and expansion, so please do not get too married to some of the routines here. I am mostly just exposing my internal commands, so things are a little ugly/hacked. I expect duplicate and alternate groups to get some form of official identifier in future, which may end up being the way to refer and edit things here.
-Also, at least for now, 'Manage File Relationships' permission is not going to be bound by the search permission restrictions that normal file search does. Getting this permission allows you to search anything. I expect to add this permission filtering tech in future, particularly for file domains.
+Also, at least for now, 'Manage File Relationships' permission is not going to be bound by the search permission restrictions that normal file search does. Getting this file relationship management permission allows you to search anything.
_There is more work to do here, including adding various 'dissolve'/'undo' commands to break groups apart._
@@ -1680,10 +1690,8 @@ Required Headers: n/a
Arguments (in percent-encoded JSON):
:
- * `file_id`: (selective, a numerical file id)
- * `file_ids`: (selective, a list of numerical file ids)
- * `hash`: (selective, a hexadecimal SHA256 hash)
- * `hashes`: (selective, a list of hexadecimal SHA256 hashes)
+ * [files](#parameters_files)
+ * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
``` title="Example request"
/manage_file_relationships/get_file_relationships?hash=ac940bb9026c430ea9530b4f4f6980a12d9432c2af8d9d39dfc67b05d91df11d
@@ -1714,7 +1722,9 @@ Response:
`is_king` and `king` relate to which file is the set best of a group. The king is usually the best representative of a group if you need to do comparisons between groups, and the 'get some pairs to filter'-style commands usually try to select the kings of the various to-be-compared duplicate groups.
-**It is possible for the king to not be available, in which case `king` is null.** The king can be unavailable in several duplicate search contexts, generally when you have the option to search/filter and it is outside of that domain. For this request, the king will usually be available unless the user has deleted it. You have to deal with the king being unavailable--in this situation, your best bet is to just use the file itself as its own representative.
+The relationships you get are filtered by the file domain. If you set the file domain to 'all known files', you will get every relationship a file has, including all deleted files, which is often less useful than you would think. The default, 'all my files' is usually most useful.
+
+**It is possible for the king to not be available, in which case `king` is null.** The king can be unavailable in several duplicate search contexts, generally when it is outside of the set file domain. For the default domain, 'all my files', the king will be available unless the user has deleted it. You have to deal with the king being unavailable--in this situation, your best bet is to just use the file itself as its own representative.
A file that has no duplicates is considered to be in a duplicate group of size 1 and thus is always its own king.
@@ -1740,6 +1750,7 @@ Required Headers: n/a
Arguments (in percent-encoded JSON):
:
+ * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
* `tag_service_key_1`: (optional, default 'all known tags', a hex tag service key)
* `tags_1`: (optional, default system:everything, a list of tags you wish to search for)
* `tag_service_key_2`: (optional, default 'all known tags', a hex tag service key)
@@ -1749,10 +1760,10 @@ Arguments (in percent-encoded JSON):
* `max_hamming_distance`: (optional, integer, default 4, the max 'search distance' of the pairs)
``` title="Example request"
-/manage_file_relationships/get_potentials_count?tag_service_key_1=c1ba23c60cda1051349647a151321d43ef5894aacdfb4b4e333d6c4259d56c5f&tags_1=%5B%22dupes_to_process%22%2C%20%22system%3Awidth%3C400%22%5D&search_type=1&pixel_duplicates=2&max_hamming_distance=0&max_num_pairs=50
+/manage_file_relationships/get_potentials_count?tag_service_key_1=c1ba23c60cda1051349647a151321d43ef5894aacdfb4b4e333d6c4259d56c5f&tags_1=%5B%22dupes_to_process%22%2C%20%22system%3Awidth%3C400%22%5D&potentials_search_type=1&pixel_duplicates=2&max_hamming_distance=0&max_num_pairs=50
```
-`tag_service_key` and `tags` work the same as [/get\_files/search\_files](#get_files_search_files). The `_2` variants are only useful if the `potentials_search_type` is 2. For now the file domain is locked to 'all my files'.
+`tag_service_key_x` and `tags_x` work the same as [/get\_files/search\_files](#get_files_search_files). The `_2` variants are only useful if the `potentials_search_type` is 2.
`potentials_search_type` and `pixel_duplicates` are enums:
@@ -1789,6 +1800,7 @@ Required Headers: n/a
Arguments (in percent-encoded JSON):
:
+ * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
* `tag_service_key_1`: (optional, default 'all known tags', a hex tag service key)
* `tags_1`: (optional, default system:everything, a list of tags you wish to search for)
* `tag_service_key_2`: (optional, default 'all known tags', a hex tag service key)
@@ -1799,7 +1811,7 @@ Arguments (in percent-encoded JSON):
* `max_num_pairs`: (optional, integer, defaults to client's option, how many pairs to get in a batch)
``` title="Example request"
-/manage_file_relationships/get_potential_pairs?tag_service_key_1=c1ba23c60cda1051349647a151321d43ef5894aacdfb4b4e333d6c4259d56c5f&tags_1=%5B%22dupes_to_process%22%2C%20%22system%3Awidth%3C400%22%5D&search_type=1&pixel_duplicates=2&max_hamming_distance=0&max_num_pairs=50
+/manage_file_relationships/get_potential_pairs?tag_service_key_1=c1ba23c60cda1051349647a151321d43ef5894aacdfb4b4e333d6c4259d56c5f&tags_1=%5B%22dupes_to_process%22%2C%20%22system%3Awidth%3C400%22%5D&potentials_search_type=1&pixel_duplicates=2&max_hamming_distance=0&max_num_pairs=50
```
The search arguments work the same as [/manage\_file\_relationships/get\_potentials\_count](#manage_file_relationships_get_potentials_count).
@@ -1833,6 +1845,7 @@ Required Headers: n/a
Arguments (in percent-encoded JSON):
:
+ * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
* `tag_service_key_1`: (optional, default 'all known tags', a hex tag service key)
* `tags_1`: (optional, default system:everything, a list of tags you wish to search for)
* `tag_service_key_2`: (optional, default 'all known tags', a hex tag service key)
@@ -1842,7 +1855,7 @@ Arguments (in percent-encoded JSON):
* `max_hamming_distance`: (optional, integer, default 4, the max 'search distance' of the files)
``` title="Example request"
-/manage_file_relationships/get_random_potentials?tag_service_key_1=c1ba23c60cda1051349647a151321d43ef5894aacdfb4b4e333d6c4259d56c5f&tags_1=%5B%22dupes_to_process%22%2C%20%22system%3Awidth%3C400%22%5D&search_type=1&pixel_duplicates=2&max_hamming_distance=0
+/manage_file_relationships/get_random_potentials?tag_service_key_1=c1ba23c60cda1051349647a151321d43ef5894aacdfb4b4e333d6c4259d56c5f&tags_1=%5B%22dupes_to_process%22%2C%20%22system%3Awidth%3C400%22%5D&potentials_search_type=1&pixel_duplicates=2&max_hamming_distance=0
```
The arguments work the same as [/manage\_file\_relationships/get\_potentials\_count](#manage_file_relationships_get_potentials_count), with the caveat that `potentials_search_type` has special logic:
@@ -1881,13 +1894,20 @@ Required Headers:
Arguments (in JSON):
:
- * `pair_rows`: (a list of lists)
+ * `relationships`: (a list of Objects, one for each file-pair being set)
-Each row is:
+Each Object is:
- * [ relationship, hash_a, hash_b, do_default_content_merge, delete_a, delete_b ]
+ * `hash_a`: (a hexadecimal SHA256 hash)
+ * `hash_b`: (a hexadecimal SHA256 hash)
+ * `relationship`: (integer enum for the relationship being set)
+ * `do_default_content_merge`: (bool)
+ * `delete_a`: (optional, bool, default false)
+ * `delete_b`: (optional, bool, default false)
-Where `relationship` is one of this enum:
+`hash_a` and `hash_b` are normal hex SHA256 hashes for your file pair.
+
+`relationship` is one of this enum:
* 0 - set as potential duplicates
* 1 - set as false positives
@@ -1898,18 +1918,33 @@ Where `relationship` is one of this enum:
2, 4, and 7 all make the files 'duplicates' (8 under `get_file_relationships`), which, specifically, merges the two files' duplicate groups. 'same quality' has different duplicate content merge options to the better/worse choices, but it ultimately sets A>B. You obviously don't have to use 'B is better' if you prefer just to swap the hashes. Do what works for you.
-`hash_a` and `hash_b` are normal hex SHA256 hashes for your file pair.
+`do_default_content_merge` sets whether the user's duplicate content merge options should be loaded and applied to the files along with the relationship. Most operations in the client do this automatically, so the user may expect it to apply, but if you want to do content merge yourself, set this to false.
-`do_default_content_merge` is a boolean setting whether the user's duplicate content merge options should be loaded and applied to the files along with the duplicate status. Most operations in the client do this automatically, so the user may expect it to apply, but if you want to do content merge yourself, set this to false.
-
-`delete_a` and `delete_b` are booleans that obviously select whether to delete A and/or B. You can also do this externally if you prefer.
+`delete_a` and `delete_b` are booleans that select whether to delete A and/or B in the same operation as setting the relationship. You can also do this externally if you prefer.
```json title="Example request body"
{
- "pair_rows" : [
- [ 4, "b54d09218e0d6efc964b78b070620a1fa19c7e069672b4c6313cee2c9b0623f2", "bbaa9876dab238dcf5799bfd8319ed0bab805e844f45cf0de33f40697b11a845", true, false, true ],
- [ 4, "22667427eaa221e2bd7ef405e1d2983846c863d40b2999ce8d1bf5f0c18f5fb2", "65d228adfa722f3cd0363853a191898abe8bf92d9a514c6c7f3c89cfed0bf423", true, false, true ],
- [ 2, "0480513ffec391b77ad8c4e57fe80e5b710adfa3cb6af19b02a0bd7920f2d3ec", "5fab162576617b5c3fc8caabea53ce3ab1a3c8e0a16c16ae7b4e4a21eab168a7", true, false, false ]
+ "relationships" : [
+ {
+ "hash_a" : "b54d09218e0d6efc964b78b070620a1fa19c7e069672b4c6313cee2c9b0623f2",
+ "hash_b" : "bbaa9876dab238dcf5799bfd8319ed0bab805e844f45cf0de33f40697b11a845",
+ "relationship" : 4,
+ "do_default_content_merge" : true,
+ "delete_b" : true
+ },
+ {
+ "hash_a" : "22667427eaa221e2bd7ef405e1d2983846c863d40b2999ce8d1bf5f0c18f5fb2",
+ "hash_b" : "65d228adfa722f3cd0363853a191898abe8bf92d9a514c6c7f3c89cfed0bf423",
+ "relationship" : 4,
+ "do_default_content_merge" : true,
+ "delete_b" : true
+ },
+ {
+ "hash_a" : "0480513ffec391b77ad8c4e57fe80e5b710adfa3cb6af19b02a0bd7920f2d3ec",
+ "hash_b" : "5fab162576617b5c3fc8caabea53ce3ab1a3c8e0a16c16ae7b4e4a21eab168a7",
+ "relationship" : 2,
+ "do_default_content_merge" : true
+ }
]
}
```
@@ -1917,7 +1952,7 @@ Where `relationship` is one of this enum:
Response:
: 200 with no content.
-If you try to add an invalid or redundant relationship, for instance setting that files that are already duplicates are potential duplicates, no changes are made.
+If you try to add an invalid or redundant relationship, for instance setting files that are already duplicates as potential duplicates, no changes are made.
This is the file relationships request that is probably most likely to change in future. I may implement content merge options. I may move from file pairs to group identifiers. When I expand alternates, those file groups are going to support more variables.
@@ -1934,10 +1969,7 @@ Required Headers:
Arguments (in JSON):
:
- * `file_id`: (selective, a numerical file id)
- * `file_ids`: (selective, a list of numerical file ids)
- * `hash`: (selective, a hexadecimal SHA256 hash)
- * `hashes`: (selective, a list of hexadecimal SHA256 hashes)
+ * [files](#parameters_files)
```json title="Example request body"
{
@@ -2061,36 +2093,42 @@ Response:
"pages" : {
"name" : "top pages notebook",
"page_key" : "3b28d8a59ec61834325eb6275d9df012860a1ecfd9e1246423059bc47fb6d5bd",
+ "page_state" : 0,
"page_type" : 10,
"selected" : true,
"pages" : [
{
"name" : "files",
"page_key" : "d436ff5109215199913705eb9a7669d8a6b67c52e41c3b42904db083255ca84d",
+ "page_state" : 0,
"page_type" : 6,
"selected" : false
},
{
"name" : "thread watcher",
"page_key" : "40887fa327edca01e1d69b533dddba4681b2c43e0b4ebee0576177852e8c32e7",
+ "page_state" : 0,
"page_type" : 9,
"selected" : false
},
{
"name" : "pages",
"page_key" : "2ee7fa4058e1e23f2bd9e915cdf9347ae90902a8622d6559ba019a83a785c4dc",
+ "page_state" : 0,
"page_type" : 10,
"selected" : true,
"pages" : [
{
"name" : "urls",
"page_key" : "9fe22cb760d9ee6de32575ed9f27b76b4c215179cf843d3f9044efeeca98411f",
+ "page_state" : 0,
"page_type" : 7,
"selected" : true
},
{
"name" : "files",
"page_key" : "2977d57fc9c588be783727bcd54225d577b44e8aa2f91e365a3eb3c3f580dc4e",
+ "page_state" : 0,
"page_type" : 6,
"selected" : false
}
@@ -2101,7 +2139,11 @@ Response:
}
```
- The page types are as follows:
+ `name` is the full text on the page tab.
+
+ `page_key` is a unique identifier for the page. It will stay the same for a particular page throughout the session, but new ones are generated on a session reload.
+
+ `page_type` is as follows:
* 1 - Gallery downloader
* 2 - Simple downloader
@@ -2112,10 +2154,20 @@ Response:
* 8 - Duplicates
* 9 - Thread watcher
* 10 - Page of pages
-
- The top page of pages will always be there, and always selected. 'selected' means which page is currently in view and will propagate down other page of pages until it terminates. It may terminate in an empty page of pages, so do not assume it will end on a 'media' page.
-
- The 'page_key' is a unique identifier for the page. It will stay the same for a particular page throughout the session, but new ones are generated on a client restart or other session reload.
+
+ `page_state` is as follows:
+
+ * 0 - ready
+ * 1 - initialising
+ * 2 - searching/loading
+ * 3 - search cancelled
+
+ Most pages will be 0, normal/ready, at all times. Large pages will start in an 'initialising' state for a few seconds, which means their session-saved thumbnails aren't loaded yet. Search pages will enter 'searching' after a refresh or search change and will either return to 'ready' when the search is complete, or fall to 'search cancelled' if the search was interrupted (usually this means the user clicked the 'stop' button that appears after some time).
+
+ `selected` means which page is currently in view. It will propagate down the page of pages until it terminates. It may terminate in an empty page of pages, so do not assume it will end on a media page.
+
+ The top page of pages will always be there, and always selected.
+
### **GET `/manage_pages/get_page_info`** { id="manage_pages_get_page_info" }
@@ -2145,6 +2197,7 @@ Response description
"page_info" : {
"name" : "threads",
"page_key" : "aebbf4b594e6986bddf1eeb0b5846a1e6bc4e07088e517aff166f1aeb1c3c9da",
+ "page_state" : 0,
"page_type" : 3,
"management" : {
"multiple_watcher_import" : {
@@ -2206,6 +2259,8 @@ Response description
}
```
+ `name`, `page_key`, `page_state`, and `page_type` are as in [/manage\_pages/get\_pages](#manage_pages_get_pages).
+
As you can see, even the 'simple' mode can get very large. Imagine that response for a page watching 100 threads! Turning simple mode off will display every import item, gallery log entry, and all hashes in the media (thumbnail) panel.
For this first version, the five importer pages--hdd import, simple downloader, url downloader, gallery page, and watcher page--all give rich info based on their specific variables. The first three only have one importer/gallery log combo, but the latter two of course can have multiple. The "imports" and "gallery_log" entries are all in the same data format.
@@ -2225,10 +2280,7 @@ Required Headers:
Arguments (in JSON):
:
* `page_key`: (the page key for the page you wish to add files to)
- * `file_id`: (selective, a numerical file id)
- * `file_ids`: (selective, a list of numerical file ids)
- * `hash`: (selective, a hexadecimal SHA256 hash)
- * `hashes`: (selective, a list of hexadecimal SHA256 hashes)
+ * [files](#parameters_files)
The files you set will be appended to the given page, just like a thumbnail drag and drop operation. The page key is the same as fetched in the [/manage\_pages/get\_pages](#manage_pages_get_pages) call.
@@ -2295,6 +2347,8 @@ The page key is the same as fetched in the [/manage\_pages/get\_pages](#manage_p
Response:
: 200 with no content. If the page key is not found, this will 404.
+Poll the `page_state` in [/manage\_pages/get\_pages](#manage_pages_get_pages) or [/manage\_pages/get\_page\_info](#manage_pages_get_page_info) to see when the search is complete.
+
## Managing the Database
### **POST `/manage_database/lock_on`** { id="manage_database_lock_on" }
diff --git a/docs/developer_api_future.md b/docs/developer_api_future.md
deleted file mode 100644
index 235b0670..00000000
--- a/docs/developer_api_future.md
+++ /dev/null
@@ -1,2318 +0,0 @@
----
-title: API documentation
-hide: navigation
----
-
-# API documentation
-
-## Library modules created by hydrus users
-
-* [Hydrus API](https://gitlab.com/cryzed/hydrus-api): A python module that talks to the API.
-* [hydrus.js](https://github.com/cravxx/hydrus.js): A node.js module that talks to the API.
-* [more projects on github](https://github.com/stars/hydrusnetwork/lists/hydrus-related-projects)
-
-## API
-
-In general, the API deals with standard UTF-8 JSON. POST requests and 200 OK responses are generally going to be a JSON 'Object' with variable names as keys and values obviously as values. There are examples throughout this document. For GET requests, everything is in standard GET parameters, but some variables are complicated and will need to be JSON encoded and then URL encoded. An example would be the 'tags' parameter on [GET /get\_files/search\_files](#get_files_search_files), which is a list of strings. Since GET http URLs have limits on what characters are allowed, but hydrus tags can have all sorts of characters, you'll be doing this:
-
-* Your list of tags:
-
- ```
- [ 'character:samus aran', 'creator:青い桜', 'system:height > 2000' ]
- ```
-
-* JSON encoded:
-
- ```json
- ["character:samus aran", "creator:\\u9752\\u3044\\u685c", "system:height > 2000"]
- ```
-
-* Then URL encoded:
-
- ```
- %5B%22character%3Asamus%20aran%22%2C%20%22creator%3A%5Cu9752%5Cu3044%5Cu685c%22%2C%20%22system%3Aheight%20%3E%202000%22%5D
- ```
-
-* In python, converting your tag list to the URL encoded string would be:
-
- ```
- urllib.parse.quote( json.dumps( tag_list ) )
- ```
-
-* Full URL path example:
-
- ```
- /get_files/search_files?file_sort_type=6&file_sort_asc=false&tags=%5B%22character%3Asamus%20aran%22%2C%20%22creator%3A%5Cu9752%5Cu3044%5Cu685c%22%2C%20%22system%3Aheight%20%3E%202000%22%5D
- ```
-
-
-On 200 OK, the API returns JSON for everything except actual file/thumbnail requests. On 4XX and 5XX, assume it will return plain text, which may be a raw traceback that I'd be interested in seeing. You'll typically get 400 for a missing parameter, 401/403/419 for missing/insufficient/expired access, and 500 for a real deal serverside error.
-
-!!! note
- For any request sent to the API, the total size of the initial request line (this includes the URL and any parameters) and the headers must not be larger than 2 megabytes.
- Exceeding this limit will cause the request to fail. Make sure to use pagination if you are passing very large JSON arrays as parameters in a GET request.
-
-
-## CBOR
-
-The API now tentatively supports CBOR, which is basically 'byte JSON'. If you are in a lower level language or need to do a lot of heavy work quickly, try it out!
-
-To send CBOR, for POST put Content-Type `application/cbor` in your request header instead of `application/json`, and for GET just add a `cbor=1` parameter to the URL string. Use CBOR to encode any parameters that you would previously put in JSON:
-
-For POST requests, just print the pure bytes in the body, like this:
-
-```
-cbor2.dumps( arg_dict )
-```
-
-For GET, encode the parameter value in base64, like this:
-
-```
-base64.urlsafe_b64encode( cbor2.dumps( argument ) )
-```
--or-
-```
-str( base64.urlsafe_b64encode( cbor2.dumps( argument ) ), 'ascii' )
-```
-
-If you send CBOR, the client will return CBOR. If you want to send CBOR and get JSON back, or _vice versa_ (or you are uploading a file and can't set CBOR Content-Type), send the Accept request header, like so:
-
-```
-Accept: application/cbor
-Accept: application/json
-```
-
-If the client does not support CBOR, you'll get 406.
-
-## Access and permissions
-
-The client gives access to its API through different 'access keys', which are the typical 64-character hex used in many other places across hydrus. Each guarantees different permissions such as handling files or tags. Most of the time, a user will provide full access, but do not assume this. If the access header or parameter is not provided, you will get 401, and all insufficient permission problems will return 403 with appropriate error text.
-
-Access is required for every request. You can provide this as an http header, like so:
-
-```
-Hydrus-Client-API-Access-Key : 0150d9c4f6a6d2082534a997f4588dcf0c56dffe1d03ffbf98472236112236ae
-```
-
-Or you can include it as a GET or POST parameter on any request (except _POST /add\_files/add\_file_, which uses the entire POST body for the file's bytes). Use the same name for your GET or POST argument, such as:
-
-```
-/get_files/thumbnail?file_id=452158&Hydrus-Client-API-Access-Key=0150d9c4f6a6d2082534a997f4588dcf0c56dffe1d03ffbf98472236112236ae
-```
-
-There is now a simple 'session' system, where you can get a temporary key that gives the same access without having to include the permanent access key in every request. You can fetch a session key with the [/session_key](#session_key) command and thereafter use it just as you would an access key, just with _Hydrus-Client-API-Session-Key_ instead.
-
-Session keys will expire if they are not used within 24 hours, or if the client is restarted, or if the underlying access key is deleted. An invalid/expired session key will give a **419** result with an appropriate error text.
-
-Bear in mind the Client API is still under construction. Setting up the Client API to be accessible across the internet requires technical experience to be convenient. HTTPS is available for encrypted comms, but the default certificate is self-signed (which basically means an eavesdropper can't see through it, but your ISP/government could if they decided to target you). If you have your own domain and SSL cert, you can replace them though (check the db directory for client.crt and client.key). Otherwise, be careful about transmitting sensitive content outside of your localhost/network.
-
-## Common Complex Parameters
-
-### **files** { id="parameters_files" }
-
-If you need to refer to some files, you can use any of the following:
-
-Arguments:
-:
- * `file_id`: (selective, a numerical file id)
- * `file_ids`: (selective, a list of numerical file ids)
- * `hash`: (selective, a hexadecimal SHA256 hash)
- * `hashes`: (selective, a list of hexadecimal SHA256 hashes)
-
-In GET requests, make sure any list is percent-encoded.
-
-### **file domain** { id="parameters_file_domain" }
-
-When you are searching, you may want to specify a particular file domain. Most of the time, you'll want to just set `file_service_key`, but this can get complex:
-
-Arguments:
-:
- * `file_service_key`: (optional, selective A, hexadecimal, the file domain on which to search)
- * `file_service_keys`: (optional, selective A, list of hexadecimals, the union of file domains on which to search)
- * `deleted_file_service_key`: (optional, selective B, hexadecimal, the 'deleted from this file domain' on which to search)
- * `deleted_file_service_keys`: (optional, selective B, list of hexadecimals, the union of 'deleted from this file domain' on which to search)
-
-The service keys are as in [/get_services](#get_services).
-
-Hydrus supports two concepts here:
-
-* Searching over a UNION of subdomains. If the user has several local file domains, e.g. 'favourites', 'personal', 'sfw', and 'nsfw', they might like to search two of them at once.
-* Searching deleted files of subdomains. You can specifically, and quickly, search the files that have been deleted from somewhere.
-
-You can play around with this yourself by clicking 'multiple locations' in the client with _help->advanced mode_ on.
-
-In extreme edge cases, these two can be mixed by populating both A and B selective, making a larger union of both current and deleted file records.
-
-Please note that unions can be very very computationally expensive. If you can achieve what you want with a single file_service_key, two queries in a row with different service keys, or an umbrella like `all my files` or `all local files`, please do. Otherwise, let me know what is running slow and I'll have a look at it.
-
-'deleted from all local files' includes all files that have been physically deleted (i.e. deleted from the trash) and not available any more for fetch file/thumbnail requests. 'deleted from all my files' includes all of those physically deleted files _and_ the trash. If a file is deleted with the special 'do not leave a deletion record' command, then it won't show up in a 'deleted from file domain' search!
-
-'all known files' is a tricky domain. It converts much of the search tech to ignore where files actually are and look at the accompanying tag domain (e.g. all the files that have been tagged), and can sometimes be very expensive.
-
-Also, if you have the option to set both file and tag domains, you cannot enter 'all known files'/'all known tags'. It is too complicated to support, sorry!
-
-## Access Management
-
-### **GET `/api_version`** { id="api_version" }
-
-_Gets the current API version. I will increment this every time I alter the API._
-
-Restricted access: NO.
-
-Required Headers: n/a
-
-Arguments: n/a
-
-Response:
-: Some simple JSON describing the current api version (and hydrus client version, if you are interested).
-: Note that this is mostly obselete now, since the 'Server' header of every response (and a duplicated 'Hydrus-Server' one, if you have a complicated proxy situation that overwrites 'Server') are now in the form "client api/{client_api_version} ({software_version})", e.g. "client api/32 (497)".
-
-```json title="Example response"
-{
- "version" : 17,
- "hydrus_version" : 441
-}
-```
-
-### **GET `/request_new_permissions`** { id="request_new_permissions" }
-
-_Register a new external program with the client. This requires the 'add from api request' mini-dialog under_ services->review services _to be open, otherwise it will 403._
-
-Restricted access: NO.
-
-Required Headers: n/a
-
-Arguments:
-
-: * `name`: (descriptive name of your access)
- * `basic_permissions`: A JSON-encoded list of numerical permission identifiers you want to request.
-
- The permissions are currently:
-
- * 0 - Import and Edit URLs
- * 1 - Import and Delete Files
- * 2 - Edit File Tags
- * 3 - Search for and Fetch Files
- * 4 - Manage Pages
- * 5 - Manage Cookies
- * 6 - Manage Database
- * 7 - Edit File Notes
- * 8 - Manage File Relationships
-
- ``` title="Example request"
- /request_new_permissions?name=my%20import%20script&basic_permissions=[0,1]
- ```
-
-Response:
-: Some JSON with your access key, which is 64 characters of hex. This will not be valid until the user approves the request in the client ui.
-```json title="Example response"
-{"access_key" : "73c9ab12751dcf3368f028d3abbe1d8e2a3a48d0de25e64f3a8f00f3a1424c57"}
-```
-
-### **GET `/session_key`** { id="session_key" }
-
-_Get a new session key._
-
-Restricted access: YES. No permissions required.
-
-Required Headers: n/a
-
-Arguments: n/a
-
-Response:
-: Some JSON with a new session key in hex.
-```json title="Example response"
-{
- "session_key" : "f6e651e7467255ade6f7c66050f3d595ff06d6f3d3693a3a6fb1a9c2b278f800"
-}
-```
-
-!!! note
- Note that the access you provide to get a new session key **can** be a session key, if that happens to be useful. As long as you have some kind of access, you can generate a new session key.
-
- A session key expires after 24 hours of inactivity, whenever the client restarts, or if the underlying access key is deleted. A request on an expired session key returns 419.
-
-
-### **GET `/verify_access_key`** { id="verify_access_key" }
-
-_Check your access key is valid._
-
-Restricted access: YES. No permissions required.
-
-Required Headers: n/a
-
-Arguments: n/a
-
-Response:
-: 401/403/419 and some error text if the provided access/session key is invalid, otherwise some JSON with basic permission info.
-```json title="Example response"
-{
- "basic_permissions" : [0, 1, 3],
- "human_description" : "API Permissions (autotagger): add tags to files, import files, search for files: Can search: only autotag this"
-}
-```
-
-
-### **GET `/get_services`** { id="get_services" }
-
-_Ask the client about its file and tag services._
-
-Restricted access:
-: YES. At least one of Add Files, Add Tags, Manage Pages, or Search Files permission needed.
-
-Required Headers: n/a
-
-Arguments: n/a
-
-Response:
-: Some JSON listing the client's file and tag services by name and 'service key'.
-```json title="Example response"
-{
- "local_tags" : [
- {
- "name" : "my tags",
- "service_key" : "6c6f63616c2074616773",
- "type" : 5,
- "type_pretty" : "local tag service"
- },
- {
- "name" : "filenames",
- "service_key" : "231a2e992b67101318c410abb6e7d98b6e32050623f138ca93bd4ad2993de31b",
- "type" : 5,
- "type_pretty" : "local tag service"
- }
- ],
- "tag_repositories" : [
- {
- "name" : "PTR",
- "service_key" : "ccb0cf2f9e92c2eb5bd40986f72a339ef9497014a5fb8ce4cea6d6c9837877d9",
- "type" : 0,
- "type_pretty" : "hydrus tag repository"
- }
- ],
- "file_repositories" : [
- {
- "name" : "kamehameha central",
- "service_key" : "89295dc26dae3ea7d395a1746a8fe2cb836b9472b97db48024bd05587f32ab0b",
- "type" : 1,
- "type_pretty" : "hydrus file repository"
- }
- ],
- "local_files" : [
- {
- "name" : "my files",
- "service_key" : "6c6f63616c2066696c6573",
- "type" : 2,
- "type_pretty" : "local file domain"
- }
- ],
- "all_local_media" : [
- {
- "name" : "all my files",
- "service_key" : "616c6c206c6f63616c206d65646961",
- "type" : 21,
- "type_pretty" : "virtual combined local media service"
- }
- ],
- "trash" : [
- {
- "name" : "trash",
- "service_key" : "7472617368",
- "type" : 14,
- "type_pretty" : "local trash file domain"
- }
- ],
- "local_updates" : [
- {
- "name" : "repository updates",
- "service_key" : "7265706f7369746f72792075706461746573",
- "type" : 20,
- "type_pretty" : "local update file domain"
- }
- ],
- "all_local_files" : [
- {
- "name" : "all local files",
- "service_key" : "616c6c206c6f63616c2066696c6573",
- "type" : 15,
- "type_pretty" : "virtual combined local file service"
- }
- ],
- "all_known_files" : [
- {
- "name" : "all known files",
- "service_key" : "616c6c206b6e6f776e2066696c6573",
- "type" : 11,
- "type_pretty" : "virtual combined file service"
- }
- ],
- "all_known_tags" : [
- {
- "name" : "all known tags",
- "service_key" : "616c6c206b6e6f776e2074616773",
- "type" : 10,
- "type_pretty" : "virtual combined tag service"
- }
- ]
-}
-```
- These services may be referred to in various metadata responses or required in request parameters (e.g. where to add tag mappings). Note that a user can rename their services. The older parts of the Client API use the renameable 'service name' as service identifier, but wish to move away from this. Please use the hex 'service_key', which is a non-mutable ID specific to each client. The hardcoded services have shorter service key strings (it is usually just 'all known files' etc.. ASCII-converted to hex), but user-made stuff will have 64-character hex.
-
- Now that I state `type` and `type_pretty` here, I may rearrange this call, probably to make the `service_key` the Object key, rather than the arbitrary 'all_known_tags' strings.
-
- You won't see all these, and you'll only ever need some, but `type` is:
-
- * 0 - tag repository
- * 1 - file repository
- * 2 - a local file domain like 'my files'
- * 5 - a local tag domain like 'my tags'
- * 6 - a 'numerical' rating service with several stars
- * 7 - a 'like/dislike' rating service with on/off status
- * 10 - all known tags -- a union of all the tag services
- * 11 - all known files -- a union of all the file services and files that appear in tag services
- * 12 - the local booru -- you can ignore this
- * 13 - IPFS
- * 14 - trash
- * 15 - all local files -- all files on hard disk ('all my files' + updates + trash)
- * 17 - file notes
- * 18 - Client API
- * 19 - all deleted files -- you can ignore this
- * 20 - local updates -- a file domain to store repository update files in
- * 21 - all my files -- union of all local file domains
- * 99 - server administration
-
-
-## Importing and Deleting Files
-
-### **POST `/add_files/add_file`** { id="add_files_add_file" }
-
-_Tell the client to import a file._
-
-Restricted access:
-: YES. Import Files permission needed.
-
-Required Headers:
-: - Content-Type: `application/json` (if sending path), `application/octet-stream` (if sending file)
-
-Arguments (in JSON):
-: - `path`: (the path you want to import)
-
-```json title="Example request body"
-{"path" : "E:\\to_import\\ayanami.jpg"}
-```
-
-Arguments (as bytes):
-: You can alternately just send the file's bytes as the POST body.
-
-Response:
-: Some JSON with the import result. Please note that file imports for large files may take several seconds, and longer if the client is busy doing other db work, so make sure your request is willing to wait that long for the response.
-```json title="Example response"
-{
- "status" : 1,
- "hash" : "29a15ad0c035c0a0e86e2591660207db64b10777ced76565a695102a481c3dd1",
- "note" : ""
-}
-```
-
- `status` is:
-
- * 1 - File was successfully imported
- * 2 - File already in database
- * 3 - File previously deleted
- * 4 - File failed to import
- * 7 - File vetoed
-
- A file 'veto' is caused by the file import options (which in this case is the 'quiet' set under the client's _options->importing_) stopping the file due to its resolution or minimum file size rules, etc...
-
- 'hash' is the file's SHA256 hash in hexadecimal, and 'note' is some occasional additional human-readable text appropriate to the file status that you may recognise from hydrus's normal import workflow. For an import error, it will always be the full traceback.
-
-
-### **POST `/add_files/delete_files`** { id="add_files_delete_files" }
-
-_Tell the client to send files to the trash._
-
-Restricted access:
-: YES. Import Files permission needed.
-
-Required Headers:
-:
-* `Content-Type`: `application/json`
-
-Arguments (in JSON):
-:
-* [files](#parameters_files)
-* [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
-* `reason`: (optional, string, the reason attached to the delete action)
-
-```json title="Example request body"
-{"hash" : "78f92ba4a786225ee2a1236efa6b7dc81dd729faf4af99f96f3e20bad6d8b538"}
-```
-
-Response:
-: 200 and no content.
-
-If you specify a file service, the file will only be deleted from that location. Only local file domains are allowed (so you can't delete from a file repository or unpin from ipfs yet). It defaults to 'all my files', which will delete from all local services (i.e. force sending to trash). Sending 'all local files' on a file already in the trash will trigger a physical file delete.
-
-### **POST `/add_files/undelete_files`** { id="add_files_undelete_files" }
-
-_Tell the client to pull files back out of the trash._
-
-Restricted access:
-: YES. Import Files permission needed.
-
-Required Headers:
-:
-* `Content-Type`: application/json
-
-Arguments (in JSON):
-:
-* [files](#parameters_files)
-* [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
-
-```json title="Example request body"
-{"hash" : "78f92ba4a786225ee2a1236efa6b7dc81dd729faf4af99f96f3e20bad6d8b538"}
-```
-
-Response:
-: 200 and no content.
-
-You can use hash or hashes, whichever is more convenient.
-
-This is the reverse of a delete_files--removing files from trash and putting them back where they came from. If you specify a file service, the files will only be undeleted to there (if they have a delete record, otherwise this is nullipotent). The default, 'all my files', undeletes to all local file services for which there are deletion records. There is no error if any of the files do not currently exist in 'trash'.
-
-
-### **POST `/add_files/archive_files`** { id="add_files_archive_files" }
-
-_Tell the client to archive inboxed files._
-
-Restricted access:
-: YES. Import Files permission needed.
-
-Required Headers:
-:
-* `Content-Type`: application/json
-
-Arguments (in JSON):
-:
-* [files](#parameters_files)
-
-```json title="Example request body"
-{"hash" : "78f92ba4a786225ee2a1236efa6b7dc81dd729faf4af99f96f3e20bad6d8b538"}
-```
-
-Response:
-: 200 and no content.
-
-This puts files in the 'archive', taking them out of the inbox. It only has meaning for files currently in 'my files' or 'trash'. There is no error if any files do not currently exist or are already in the archive.
-
-
-### **POST `/add_files/unarchive_files`** { id="add_files_unarchive_files" }
-
-_Tell the client re-inbox archived files._
-
-Restricted access:
-: YES. Import Files permission needed.
-
-Required Headers:
-:
-* `Content-Type`: application/json
-
-Arguments (in JSON):
-:
-* [files](#parameters_files)
-
-```json title="Example request body"
-{"hash" : "78f92ba4a786225ee2a1236efa6b7dc81dd729faf4af99f96f3e20bad6d8b538"}
-```
-
-Response:
-: 200 and no content.
-
-This puts files back in the inbox, taking them out of the archive. It only has meaning for files currently in 'my files' or 'trash'. There is no error if any files do not currently exist or are already in the inbox.
-
-
-## Importing and Editing URLs
-
-### **GET `/add_urls/get_url_files`** { id="add_urls_get_url_files" }
-
-_Ask the client about an URL's files._
-
-Restricted access:
-: YES. Import URLs permission needed.
-
-Required Headers: n/a
-
-Arguments:
-:
- * `url`: (the url you want to ask about)
- * `doublecheck_file_system`: true or false (optional, defaults False)
-
-Example request:
-: for URL `http://safebooru.org/index.php?page=post&s=view&id=2753608`:
- ```
- /add_urls/get_url_files?url=http%3A%2F%2Fsafebooru.org%2Findex.php%3Fpage%3Dpost%26s%3Dview%26id%3D2753608
- ```
-
-Response:
-: Some JSON which files are known to be mapped to that URL. Note this needs a database hit, so it may be delayed if the client is otherwise busy. Don't rely on this to always be fast.
-```json title="Example response"
-{
- "normalised_url" : "https://safebooru.org/index.php?id=2753608&page=post&s=view",
- "url_file_statuses" : [
- {
- "status" : 2,
- "hash" : "20e9002824e5e7ffc240b91b6e4a6af552b3143993c1778fd523c30d9fdde02c",
- "note" : "url recognised: Imported at 2015/10/18 10:58:01, which was 3 years 4 months ago (before this check)."
- }
- ]
-}
-```
-
-The `url_file_statuses` is a list of zero-to-n JSON Objects, each representing a file match the client found in its database for the URL. Typically, it will be of length 0 (for as-yet-unvisited URLs or Gallery/Watchable URLs that are not attached to files) or 1, but sometimes multiple files are given the same URL (sometimes by mistaken misattribution, sometimes by design, such as pixiv manga pages). Handling n files per URL is a pain but an unavoidable issue you should account for.
-
-`status` is the same as for `/add_files/add_file`:
-
- * 0 - File not in database, ready for import (you will only see this very rarely--usually in this case you will just get no matches)
- * 2 - File already in database
- * 3 - File previously deleted
-
-`hash` is the file's SHA256 hash in hexadecimal, and 'note' is some occasional additional human-readable text you may recognise from hydrus's normal import workflow.
-
-If you set `doublecheck_file_system` to `true`, then any result that is 'already in db' (2) will be double-checked against the actual file system. This check happens on any normal file import process, just to check for and fix missing files (if the file is missing, the status becomes 0--new), but the check can take more than a few milliseconds on an HDD or a network drive, so the default behaviour, assuming you mostly just want to spam for 'seen this before' file statuses, is to not do it.
-
-### **GET `/add_urls/get_url_info`** { id="add_urls_get_url_info" }
-
-_Ask the client for information about a URL._
-
-Restricted access:
-: YES. Import URLs permission needed.
-
-Required Headers: n/a
-
-Arguments:
-:
- * `url`: (the url you want to ask about)
-
-Example request:
-: for URL `https://8ch.net/tv/res/1846574.html`:
- ```
- /add_urls/get_url_info?url=https%3A%2F%2F8ch.net%2Ftv%2Fres%2F1846574.html
- ```
-
-Response:
-: Some JSON describing what the client thinks of the URL.
-```json title="Example response"
-{
- "normalised_url" : "https://8ch.net/tv/res/1846574.html",
- "url_type" : 4,
- "url_type_string" : "watchable url",
- "match_name" : "8chan thread",
- "can_parse" : true
-}
-```
-
- The url types are currently:
-
- * 0 - Post URL
- * 2 - File URL
- * 3 - Gallery URL
- * 4 - Watchable URL
- * 5 - Unknown URL (i.e. no matching URL Class)
-
- 'Unknown' URLs are treated in the client as direct File URLs. Even though the 'File URL' type is available, most file urls do not have a URL Class, so they will appear as Unknown. Adding them to the client will pass them to the URL Downloader as a raw file for download and import.
-
-
-### **POST `/add_urls/add_url`** { id="add_urls_add_url" }
-
-_Tell the client to 'import' a URL. This triggers the exact same routine as drag-and-dropping a text URL onto the main client window._
-
-Restricted access:
-: YES. Import URLs permission needed. Add Tags needed to include tags.
-
-Required Headers:
-:
- * `Content-Type`: `application/json`
-
-Arguments (in JSON):
-:
-
- * `url`: (the url you want to add)
- * `destination_page_key`: (optional page identifier for the page to receive the url)
- * `destination_page_name`: (optional page name to receive the url)
- * `show_destination_page`: (optional, defaulting to false, controls whether the UI will change pages on add)
- * `service_keys_to_additional_tags`: (optional, selective, tags to give to any files imported from this url)
- * `filterable_tags`: (optional tags to be filtered by any tag import options that applies to the URL)
-
-If you specify a `destination_page_name` and an appropriate importer page already exists with that name, that page will be used. Otherwise, a new page with that name will be recreated (and used by subsequent calls with that name). Make sure it that page name is unique (e.g. '/b/ threads', not 'watcher') in your client, or it may not be found.
-
-Alternately, `destination_page_key` defines exactly which page should be used. Bear in mind this page key is only valid to the current session (they are regenerated on client reset or session reload), so you must figure out which one you want using the [/manage\_pages/get\_pages](#manage_pages_get_pages) call. If the correct page_key is not found, or the page it corresponds to is of the incorrect type, the standard page selection/creation rules will apply.
-
-`show_destination_page` defaults to False to reduce flicker when adding many URLs to different pages quickly. If you turn it on, the client will behave like a URL drag and drop and select the final page the URL ends up on.
-
-`service_keys_to_additional_tags` uses the same data structure as in /add\_tags/add\_tags--service keys to a list of tags to add. You will need 'add tags' permission or this will 403. These tags work exactly as 'additional' tags work in a _tag import options_. They are service specific, and always added unless some advanced tag import options checkbox (like 'only add tags to new files') is set.
-
-filterable_tags works like the tags parsed by a hydrus downloader. It is just a list of strings. They have no inherant service and will be sent to a _tag import options_, if one exists, to decide which tag services get what. This parameter is useful if you are pulling all a URL's tags outside of hydrus and want to have them processed like any other downloader, rather than figuring out service names and namespace filtering on your end. Note that in order for a tag import options to kick in, I think you will have to have a Post URL URL Class hydrus-side set up for the URL so some tag import options (whether that is Class-specific or just the default) can be loaded at import time.
-
-```json title="Example request body"
-{
- "url" : "https://8ch.net/tv/res/1846574.html",
- "destination_page_name" : "kino zone",
- "service_keys_to_additional_tags" : {
- "6c6f63616c2074616773" : ["as seen on /tv/"]
- }
-}
-```
-```json title="Example request body"
-{
- "url" : "https://safebooru.org/index.php?page=post&s=view&id=3195917",
- "filterable_tags" : [
- "1girl",
- "artist name",
- "creator:azto dio",
- "blonde hair",
- "blue eyes",
- "breasts",
- "character name",
- "commentary",
- "english commentary",
- "formal",
- "full body",
- "glasses",
- "gloves",
- "hair between eyes",
- "high heels",
- "highres",
- "large breasts",
- "long hair",
- "long sleeves",
- "looking at viewer",
- "series:metroid",
- "mole",
- "mole under mouth",
- "patreon username",
- "ponytail",
- "character:samus aran",
- "solo",
- "standing",
- "suit",
- "watermark"
- ]
-}
-```
-
-Response:
-: Some JSON with info on the URL added.
-```json title="Example response"
-{
- "human_result_text" : "\"https://8ch.net/tv/res/1846574.html\" URL added successfully.",
- "normalised_url" : "https://8ch.net/tv/res/1846574.html"
-}
-```
-
-
-
-### **POST `/add_urls/associate_url`** { id="add_urls_associate_url" }
-
-_Manage which URLs the client considers to be associated with which files._
-
-Restricted access:
-: YES. Import URLs permission needed.
-
-Required Headers:
-:
- * `Content-Type`: `application/json`
-
-
-Arguments (in JSON):
-:
- * `url_to_add`: (optional, selective A, an url you want to associate with the file(s))
- * `urls_to_add`: (optional, selective A, a list of urls you want to associate with the file(s))
- * `url_to_delete`: (optional, selective B, an url you want to disassociate from the file(s))
- * `urls_to_delete`: (optional, selective B, a list of urls you want to disassociate from the file(s))
- * [files](#parameters_files)
-
- The single/multiple arguments work the same--just use whatever is convenient for you. Unless you really know what you are doing with URL Classes, I strongly recommend you stick to associating URLs with just one single 'hash' at a time. Multiple hashes pointing to the same URL is unusual and frequently unhelpful.
-```json title="Example request body"
-{
- "url_to_add" : "https://rule34.xxx/index.php?id=2588418&page=post&s=view",
- "hash" : "3b820114f658d768550e4e3d4f1dced3ff8db77443472b5ad93700647ad2d3ba"
-}
-```
-
-Response:
-: 200 with no content. Like when adding tags, this is safely idempotent--do not worry about re-adding URLs associations that already exist or accidentally trying to delete ones that don't.
-
-
-## Editing File Tags
-
-### **GET `/add_tags/clean_tags`** { id="add_tags_clean_tags" }
-
-_Ask the client about how it will see certain tags._
-
-Restricted access:
-: YES. Add Tags permission needed.
-
-Required Headers: n/a
-
-Arguments (in percent-encoded JSON):
-:
-* `tags`: (a list of the tags you want cleaned)
-
-Example request:
-: Given tags `#!json [ " bikini ", "blue eyes", " character : samus aran ", " :)", " ", "", "10", "11", "9", "system:wew", "-flower" ]`:
- ```
- /add_tags/clean_tags?tags=%5B%22%20bikini%20%22%2C%20%22blue%20%20%20%20eyes%22%2C%20%22%20character%20%3A%20samus%20aran%20%22%2C%20%22%3A%29%22%2C%20%22%20%20%20%22%2C%20%22%22%2C%20%2210%22%2C%20%2211%22%2C%20%229%22%2C%20%22system%3Awew%22%2C%20%22-flower%22%5D
- ```
-
-Response:
-: The tags cleaned according to hydrus rules. They will also be in hydrus human-friendly sorting order.
-```json title="Example response"
-{
- "tags" : ["9", "10", "11", " ::)", "bikini", "blue eyes", "character:samus aran", "flower", "wew"]
-}
-```
-
- Mostly, hydrus simply trims excess whitespace, but the other examples are rare issues you might run into. 'system' is an invalid namespace, tags cannot be prefixed with hyphens, and any tag starting with ':' is secretly dealt with internally as "\[no namespace\]:\[colon-prefixed-subtag\]". Again, you probably won't run into these, but if you see a mismatch somewhere and want to figure it out, or just want to sort some numbered tags, you might like to try this.
-
-
-### **GET `/add_tags/search_tags`** { id="add_tags_search_tags" }
-
-_Search the client for tags._
-
-Restricted access:
-: YES. Search for Files permission needed.
-
-Required Headers: n/a
-
-Arguments:
-:
- * `search`: (the tag text to search for, enter exactly what you would in the client UI)
- * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
- * `tag_service_key`: (optional, hexadecimal, the tag domain on which to search, defaults to 'all known tags')
- * `tag_display_type`: (optional, string, to select whether to search raw or sibling-processed tags, defaults to 'storage')
-
-The `file domain` and `tag_service_key` perform the function of the file and tag domain buttons in the client UI.
-
-The `tag_display_type` can be either `storage` (the default), which searches your file's stored tags, just as they appear in a 'manage tags' dialog, or `display`, which searches the sibling-processed tags, just as they appear in a normal file search page. In the example above, setting the `tag_display_type` to `display` could well combine the two kim possible tags and give a count of 3 or 4.
-
-'all my files'/'all known tags' works fine for most cases, but a specific tag service or 'all known files'/'tag service' can work better for editing tag repository `storage` contexts, since it provides results just for that service, and for repositories, it gives tags for all the non-local files other users have tagged.
-
-Example request:
-:
-```http title="Example request"
-/add_tags/search_tags?search=kim&tag_display_type=display
-```
-
-Response:
-: Some JSON listing the client's matching tags.
-
-:
-```json title="Example response"
-{
- "tags" : [
- {
- "value" : "series:kim possible",
- "count" : 3
- },
- {
- "value" : "kimchee",
- "count" : 2
- },
- {
- "value" : "character:kimberly ann possible",
- "count" : 1
- }
- ]
-}
-```
-
-The `tags` list will be sorted by descending count. The various rules in _tags->manage tag display and search_ (e.g. no pure `*` searches on certain services) will also be checked--and if violated, you will get 200 OK but an empty result.
-
-Note that if your client api access is only allowed to search certain tags, the results will be similarly filtered.
-
-### **POST `/add_tags/add_tags`** { id="add_tags_add_tags" }
-
-_Make changes to the tags that files have._
-
-Restricted access:
-: YES. Add Tags permission needed.
-
-Required Headers: n/a
-
-Arguments (in JSON):
-:
-* [files](#parameters_files)
-* `service_keys_to_tags`: (selective B, an Object of service keys to lists of tags to be 'added' to the files)
-* `service_keys_to_actions_to_tags`: (selective B, an Object of service keys to content update actions to lists of tags)
-
- In 'service\_keys\_to...', the keys are as in [/get_services](#get_services). You may need some selection UI on your end so the user can pick what to do if there are multiple choices.
-
- Also, you can use either '...to\_tags', which is simple and add-only, or '...to\_actions\_to\_tags', which is more complicated and allows you to remove/petition or rescind pending content.
-
- The permitted 'actions' are:
-
- * 0 - Add to a local tag service.
- * 1 - Delete from a local tag service.
- * 2 - Pend to a tag repository.
- * 3 - Rescind a pend from a tag repository.
- * 4 - Petition from a tag repository. (This is special)
- * 5 - Rescind a petition from a tag repository.
-
- When you petition a tag from a repository, a 'reason' for the petition is typically needed. If you send a normal list of tags here, a default reason of "Petitioned from API" will be given. If you want to set your own reason, you can instead give a list of \[ tag, reason \] pairs.
-
-Some example requests:
-:
-```json title="Adding some tags to a file"
-{
- "hash" : "df2a7b286d21329fc496e3aa8b8a08b67bb1747ca32749acb3f5d544cbfc0f56",
- "service_keys_to_tags" : {
- "6c6f63616c2074616773" : ["character:supergirl", "rating:safe"]
- }
-}
-```
-```json title="Adding more tags to two files"
-{
- "hashes" : [
- "df2a7b286d21329fc496e3aa8b8a08b67bb1747ca32749acb3f5d544cbfc0f56",
- "f2b022214e711e9a11e2fcec71bfd524f10f0be40c250737a7861a5ddd3faebf"
- ],
- "service_keys_to_tags" : {
- "6c6f63616c2074616773" : ["process this"],
- "ccb0cf2f9e92c2eb5bd40986f72a339ef9497014a5fb8ce4cea6d6c9837877d9" : ["creator:dandon fuga"]
- }
-}
-```
-```json title="A complicated transaction with all possible actions"
-{
- "hash" : "df2a7b286d21329fc496e3aa8b8a08b67bb1747ca32749acb3f5d544cbfc0f56",
- "service_keys_to_actions_to_tags" : {
- "6c6f63616c2074616773" : {
- "0" : ["character:supergirl", "rating:safe"],
- "1" : ["character:superman"]
- },
- "aa0424b501237041dab0308c02c35454d377eebd74cfbc5b9d7b3e16cc2193e9" : {
- "2" : ["character:supergirl", "rating:safe"],
- "3" : ["filename:image.jpg"],
- "4" : [["creator:danban faga", "typo"], ["character:super_girl", "underscore"]],
- "5" : ["skirt"]
- }
- }
-}
-```
-
- This last example is far more complicated than you will usually see. Pend rescinds and petition rescinds are not common. Petitions are also quite rare, and gathering a good petition reason for each tag is often a pain.
-
- Note that the enumerated status keys in the service\_keys\_to\_actions\_to_tags structure are strings, not ints (JSON does not support int keys for Objects).
-
-Response description:
-: 200 and no content.
-
-!!! note
- Note also that hydrus tag actions are safely idempotent. You can pend a tag that is already pended, or add a tag that already exists, and not worry about an error--the surplus add action will be discarded. The same is true if you try to pend a tag that actually already exists, or rescinding a petition that doesn't. Any invalid actions will fail silently.
-
- It is fine to just throw your 'process this' tags at every file import and not have to worry about checking which files you already added them to.
-
-!!! danger "HOWEVER"
- When you delete a tag, a deletion record is made _even if the tag does not exist on the file_. This is important if you expect to add the tags again via parsing, because, in general, when hydrus adds tags through a downloader, it will not overwrite a previously 'deleted' tag record (this is to stop re-downloads overwriting the tags you hand-removed previously). Undeletes usually have to be done manually by a human.
-
- So, _do_ be careful about how you spam delete unless it is something that doesn't matter or it is something you'll only be touching again via the API anyway.
-
-## Editing File Notes
-
-### **POST `/add_notes/set_notes`** { id="add_notes_set_notes" }
-
-_Add or update notes associated with a file._
-
-Restricted access:
-: YES. Add Notes permission needed.
-
-Required Headers:
-:
- * `Content-Type`: `application/json`
-
-Arguments (in percent-encoded JSON):
-:
-* `notes`: (an Object mapping string names to string texts)
-* `hash`: (selective, an SHA256 hash for the file in 64 characters of hexadecimal)
-* `file_id`: (selective, the integer numerical identifier for the file)
-* `merge_cleverly`: true or false (optional, defaults false)
-* `extend_existing_note_if_possible`: true or false (optional, defaults true)
-* `conflict_resolution`: 0, 1, 2, or 3 (optional, defaults 3)
-
-With `merge_cleverly` left `false`, then this is a simple update operation. Existing notes will be overwritten exactly as you specify. Any other notes the file has will be untouched.
-```json title="Example request body"
-{
- "notes" : {
- "note name" : "content of note",
- "another note" : "asdf"
- },
- "hash" : "3b820114f658d768550e4e3d4f1dced3ff8db77443472b5ad93700647ad2d3ba"
-}
-```
-
-If you turn on `merge_cleverly`, then the client will merge your new notes into the file's existing notes using the same logic you have seen in Note Import Options and the Duplicate Metadata Merge Options. This navigates conflict resolution, and you should use it if you are adding potential duplicate content from an 'automatic' source like a parser and do not want to wade into the logic. Do not use it for a user-editing experience (a user expects a strict overwrite/replace experience and will be confused by this mode).
-
-To start off, in this mode, if your note text exists under a different name for the file, your dupe note will not be added to your new name. `extend_existing_note_if_possible` makes it so your existing note text will overwrite an existing name (or a '... (1)' rename of that name) if the existing text is inside your given text. `conflict_resolution` is an enum governing what to do in all other conflicts:
-
-_If a new note name already exists and its new text differs from what already exists:_
-:
-* 0 - replace - Overwrite the existing conflicting note.
-* 1 - ignore - Make no changes.
-* 2 - append - Append the new text to the existing text.
-* 3 - rename (default) - Add the new text under a 'name (x)'-style rename.
-
-Response:
-: 200 with the note changes actually sent through. If `merge_cleverly=false`, this is exactly what you gave, and this operation is idempotent. If `merge_cleverly=true`, then this may differ, even be empty, and this operation might not be idempotent.
-```json title="Example response"
-{
- "notes" : {
- "note name" : "content of note",
- "another note (1)" : "asdf"
- }
-}
-```
-
-
-### **POST `/add_notes/delete_notes`** { id="add_notes_delete_notes" }
-
-_Remove notes associated with a file._
-
-Restricted access:
-: YES. Add Notes permission needed.
-
-Required Headers:
-:
- * `Content-Type`: `application/json`
-
-Arguments (in percent-encoded JSON):
-:
-* `note_names`: (a list of string note names to delete)
-* `hash`: (selective, an SHA256 hash for the file in 64 characters of hexadecimal)
-* `file_id`: (selective, the integer numerical identifier for the file)
-
-```json title="Example request body"
-{
- "note_names" : ["note name", "another note"],
- "hash" : "3b820114f658d768550e4e3d4f1dced3ff8db77443472b5ad93700647ad2d3ba"
-}
-```
-
-Response:
-: 200 with no content. This operation is idempotent.
-
-## Searching and Fetching Files
-
-File search in hydrus is not paginated like a booru--all searches return all results in one go. In order to keep this fast, search is split into two steps--fetching file identifiers with a search, and then fetching file metadata in batches. You may have noticed that the client itself performs searches like this--thinking a bit about a search and then bundling results in batches of 256 files before eventually throwing all the thumbnails on screen.
-
-### **GET `/get_files/search_files`** { id="get_files_search_files" }
-
-_Search for the client's files._
-
-Restricted access:
-: YES. Search for Files permission needed. Additional search permission limits may apply.
-
-Required Headers: n/a
-
-Arguments (in percent-encoded JSON):
-:
- * `tags`: (a list of tags you wish to search for)
- * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
- * `tag_service_key`: (optional, hexadecimal, the tag domain on which to search, defaults to 'all my files')
- * `file_sort_type`: (optional, integer, the results sort method, defaults to 'all known tags')
- * `file_sort_asc`: true or false (optional, the results sort order)
- * `return_file_ids`: true or false (optional, default true, returns file id results)
- * `return_hashes`: true or false (optional, default false, returns hex hash results)
-
-``` title='Example request for 16 files (system:limit=16) in the inbox with tags "blue eyes", "blonde hair", and "кино"'
-/get_files/search_files?tags=%5B%22blue%20eyes%22%2C%20%22blonde%20hair%22%2C%20%22%5Cu043a%5Cu0438%5Cu043d%5Cu043e%22%2C%20%22system%3Ainbox%22%2C%20%22system%3Alimit%3D16%22%5D
-```
-
-
-If the access key's permissions only permit search for certain tags, at least one positive whitelisted/non-blacklisted tag must be in the "tags" list or this will 403. Tags can be prepended with a hyphen to make a negated tag (e.g. "-green eyes"), but these will not be checked against the permissions whitelist.
-
-Wildcards and namespace searches are supported, so if you search for 'character:sam*' or 'series:*', this will be handled correctly clientside.
-
-**Many system predicates are also supported using a text parser!** The parser was designed by a clever user for human input and allows for a certain amount of error (e.g. ~= instead of ≈, or "isn't" instead of "is not") or requires more information (e.g. the specific hashes for a hash lookup). **Here's a big list of examples that are supported:**
-
-??? example "System Predicates"
- * system:everything
- * system:inbox
- * system:archive
- * system:has duration
- * system:no duration
- * system:is the best quality file of its duplicate group
- * system:is not the best quality file of its duplicate group
- * system:has audio
- * system:no audio
- * system:has exif
- * system:no exif
- * system:has human-readable embedded metadata
- * system:no human-readable embedded metadata
- * system:has icc profile
- * system:no icc profile
- * system:has tags
- * system:no tags
- * system:untagged
- * system:number of tags > 5
- * system:number of tags ~= 10
- * system:number of tags > 0
- * system:number of words < 2
- * system:height = 600
- * system:height > 900
- * system:width < 200
- * system:width > 1000
- * system:filesize ~= 50 kilobytes
- * system:filesize > 10megabytes
- * system:filesize < 1 GB
- * system:filesize > 0 B
- * system:similar to abcdef01 abcdef02 abcdef03, abcdef04 with distance 3
- * system:similar to abcdef distance 5
- * system:limit = 100
- * system:filetype = image/jpg, image/png, apng
- * system:hash = abcdef01 abcdef02 abcdef03 _(this does sha256)_
- * system:hash = abcdef01 abcdef02 md5
- * system:modified date < 7 years 45 days 7h
- * system:modified date > 2011-06-04
- * system:last viewed time < 7 years 45 days 7h
- * system:last view time < 7 years 45 days 7h
- * system:date modified > 7 years 2 months
- * system:date modified < 0 years 1 month 1 day 1 hour
- * system:import time < 7 years 45 days 7h
- * system:time imported < 7 years 45 days 7h
- * system:time imported > 2011-06-04
- * system:time imported > 7 years 2 months
- * system:time imported < 0 years 1 month 1 day 1 hour
- * system:time imported ~= 2011-1-3
- * system:time imported ~= 1996-05-2
- * system:duration < 5 seconds
- * system:duration ~= 600 msecs
- * system:duration > 3 milliseconds
- * system:file service is pending to my files
- * system:file service currently in my files
- * system:file service is not currently in my files
- * system:file service is not pending to my files
- * system:num file relationships < 3 alternates
- * system:number of file relationships > 3 false positives
- * system:ratio is wider than 16:9
- * system:ratio is 16:9
- * system:ratio taller than 1:1
- * system:num pixels > 50 px
- * system:num pixels < 1 megapixels
- * system:num pixels ~= 5 kilopixel
- * system:media views ~= 10
- * system:all views > 0
- * system:preview views < 10
- * system:media viewtime < 1 days 1 hour 0 minutes
- * system:all viewtime > 1 hours 100 seconds
- * system:preview viewtime ~= 1 day 30 hours 100 minutes 90s
- * system:has url matching regex index\\.php
- * system:does not have a url matching regex index\\.php
- * system:has url https://safebooru.donmai.us/posts/4695284
- * system:does not have url https://safebooru.donmai.us/posts/4695284
- * system:has domain safebooru.com
- * system:does not have domain safebooru.com
- * system:has a url with class safebooru file page
- * system:does not have a url with url class safebooru file page
- * system:tag as number page < 5
- * system:has notes
- * system:no notes
- * system:does not have notes
- * system:num notes is 5
- * system:num notes > 1
- * system:has note with name note name
- * system:no note with name note name
- * system:does not have note with name note name
-
-Please test out the system predicates you want to send. If you are in _help->advanced mode_, you can test this parser in the advanced text input dialog when you click the OR\* button on a tag autocomplete dropdown. More system predicate types and input formats will be available in future. Reverse engineering system predicate data from text is obviously tricky. If a system predicate does not parse, you'll get 400.
-
-Also, OR predicates are now supported! Just nest within the tag list, and it'll be treated like an OR. For instance:
-
-* `#!json [ "skirt", [ "samus aran", "lara croft" ], "system:height > 1000" ]`
-
-Makes:
-
-* skirt
-* samus aran OR lara croft
-* system:height > 1000
-
-The file and tag services are for search domain selection, just like clicking the buttons in the client. They are optional--default is 'all my files' and 'all known tags'.
-
-File searches occur in the `display` `tag_display_type`. If you want to pair autocomplete tag lookup from [/search_tags](#add_tags_search_tags) to this file search (e.g. for making a standard booru search interface), then make sure you are searching `display` tags there.
-
-file\_sort\_asc is 'true' for ascending, and 'false' for descending. The default is descending.
-
-file\_sort\_type is by default _import time_. It is an integer according to the following enum, and I have written the semantic (asc/desc) meaning for each type after:
-
-* 0 - file size (smallest first/largest first)
-* 1 - duration (shortest first/longest first)
-* 2 - import time (oldest first/newest first)
-* 3 - filetype (N/A)
-* 4 - random (N/A)
-* 5 - width (slimmest first/widest first)
-* 6 - height (shortest first/tallest first)
-* 7 - ratio (tallest first/widest first)
-* 8 - number of pixels (ascending/descending)
-* 9 - number of tags (on the current tag domain) (ascending/descending)
-* 10 - number of media views (ascending/descending)
-* 11 - total media viewtime (ascending/descending)
-* 12 - approximate bitrate (smallest first/largest first)
-* 13 - has audio (audio first/silent first)
-* 14 - modified time (oldest first/newest first)
-* 15 - framerate (slowest first/fastest first)
-* 16 - number of frames (smallest first/largest first)
-* 18 - last viewed time (oldest first/newest first)
-* 19 - archive timestamp (oldest first/newest first)
-* 20 - hash hex (N/A)
-
-Response:
-: The full list of numerical file ids that match the search.
-```json title="Example response"
-{
- "file_ids" : [125462, 4852415, 123, 591415]
-}
-```
-```json title="Example response with return_hashes=true"
-{
- "hashes" : [
- "1b04c4df7accd5a61c5d02b36658295686b0abfebdc863110e7d7249bba3f9ad",
- "fe416723c731d679aa4d20e9fd36727f4a38cd0ac6d035431f0f452fad54563f",
- "b53505929c502848375fbc4dab2f40ad4ae649d34ef72802319a348f81b52bad"
- ],
- "file_ids" : [125462, 4852415, 123]
-}
-```
-
- You can of course also specify `return_hashes=true&return_file_ids=false` just to get the hashes. The order of both lists is the same.
-
- File ids are internal and specific to an individual client. For a client, a file with hash H always has the same file id N, but two clients will have different ideas about which N goes with which H. IDs are a bit faster to retrieve than hashes and search with _en masse_, which is why they are exposed here.
-
- This search does **not** apply the implicit limit that most clients set to all searches (usually 10,000), so if you do system:everything on a client with millions of files, expect to get boshed. Even with a system:limit included, complicated queries with large result sets may take several seconds to respond. Just like the client itself.
-
-### **GET `/get_files/file_hashes`** { id="get_files_file_hashes" }
-
-_Lookup file hashes from other hashes._
-
-Restricted access:
-: YES. Search for Files permission needed.
-
-Required Headers: n/a
-
-Arguments (in percent-encoded JSON):
-:
- * `hash`: (selective, a hexadecimal hash)
- * `hashes`: (selective, a list of hexadecimal hashes)
- * `source_hash_type`: [sha256|md5|sha1|sha512] (optional, defaulting to sha256)
- * `desired_hash_type`: [sha256|md5|sha1|sha512]
-
-If you have some MD5 hashes and want to see what their SHA256 are, or _vice versa_, this is the place. Hydrus records the non-SHA256 hashes for every file it has ever imported. This data is not removed on file deletion.
-
-``` title="Example request"
-/get_files/file_hashes?hash=ec5c5a4d7da4be154597e283f0b6663c&source_hash_type=md5&desired_hash_type=sha256
-```
-
-Response:
-: A mapping Object of the successful lookups. Where no matching hash is found, no entry will be made (therefore, if none of your source hashes have matches on the client, this will return an empty `hashes` Object).
-```json title="Example response"
-{
- "hashes" : {
- "ec5c5a4d7da4be154597e283f0b6663c" : "2a0174970defa6f147f2eabba829c5b05aba1f1aea8b978611a07b7bb9cf9399"
- }
-}
-```
-
-### **GET `/get_files/file_metadata`** { id="get_files_file_metadata" }
-
-_Get metadata about files in the client._
-
-Restricted access:
-: YES. Search for Files permission needed. Additional search permission limits may apply.
-
-Required Headers: n/a
-
-Arguments (in percent-encoded JSON):
-:
- * [files](#parameters_files)
- * `create_new_file_ids`: true or false (optional if asking with hash(es), defaulting to false)
- * `only_return_identifiers`: true or false (optional, defaulting to false)
- * `only_return_basic_information`: true or false (optional, defaulting to false)
- * `detailed_url_information`: true or false (optional, defaulting to false)
- * `include_notes`: true or false (optional, defaulting to false)
- * `hide_service_keys_tags`: **Deprecated, will be deleted soon!** true or false (optional, defaulting to true)
-
-You need one of file_ids or hashes. If your access key is restricted by tag, you cannot search by hashes, and **the file_ids you search for must have been in the most recent search result**.
-
-``` title="Example request for two files with ids 123 and 4567"
-/get_files/file_metadata?file_ids=%5B123%2C%204567%5D
-```
-
-``` title="The same, but only wants hashes back"
-/get_files/file_metadata?file_ids=%5B123%2C%204567%5D&only_return_identifiers=true
-```
-
-``` title="And one that fetches two hashes"
-/get_files/file_metadata?hashes=%5B%224c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2%22%2C%20%223e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82%22%5D
-```
-
-This request string can obviously get pretty ridiculously long. It also takes a bit of time to fetch metadata from the database. In its normal searches, the client usually fetches file metadata in batches of 256.
-
-Response:
-: A list of JSON Objects that store a variety of file metadata.
-```json title="Example response"
-{
- "metadata" : [
- {
- "file_id" : 123,
- "hash" : "4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2",
- "size" : 63405,
- "mime" : "image/jpeg",
- "ext" : ".jpg",
- "width" : 640,
- "height" : 480,
- "thumbnail_width" : 200,
- "thumbnail_height" : 150,
- "duration" : null,
- "time_modified" : null,
- "time_modified_details" : {},
- "file_services" : {
- "current" : {},
- "deleted" : {}
- },
- "ipfs_multihashes" : {},
- "has_audio" : false,
- "num_frames" : null,
- "num_words" : null,
- "is_inbox" : false,
- "is_local" : false,
- "is_trashed" : false,
- "is_deleted" : false,
- "has_exif" : true,
- "has_human_readable_embedded_metadata" : true,
- "has_icc_profile" : true,
- "known_urls" : [],
- "tags" : {
- "6c6f63616c2074616773" : {
- "name" : "local tags",
- "type" : 5,
- "type_pretty" : "local tag service",
- "storage_tags" : {},
- "display_tags" : {}
- },
- "37e3849bda234f53b0e9792a036d14d4f3a9a136d1cb939705dbcd5287941db4" : {
- "name" : "public tag repo",
- "type" : 1,
- "type_pretty" : "hydrus tag repository",
- "storage_tags" : {},
- "display_tags" : {}
- },
- "616c6c206b6e6f776e2074616773" : {
- "name" : "all known tags",
- "type" : 10,
- "type_pretty" : "virtual combined tag service",
- "storage_tags" : {},
- "display_tags" : {}
- }
- }
- },
- {
- "file_id" : 4567,
- "hash" : "3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82",
- "size" : 199713,
- "mime" : "video/webm",
- "ext" : ".webm",
- "width" : 1920,
- "height" : 1080,
- "thumbnail_width" : 200,
- "thumbnail_height" : 113,
- "duration" : 4040,
- "time_modified" : 1604055647,
- "time_modified_details" : {
- "local" : 1641044491,
- "gelbooru.com" : 1604055647
- },
- "file_services" : {
- "current" : {
- "616c6c206c6f63616c2066696c6573" : {
- "name" : "all local files",
- "type" : 15,
- "type_pretty" : "virtual combined local file service",
- "time_imported" : 1641044491
- },
- "616c6c206c6f63616c2066696c6573" : {
- "name" : "all my files",
- "type" : 21,
- "type_pretty" : "virtual combined local media service",
- "time_imported" : 1641044491
- },
- "cb072cffbd0340b67aec39e1953c074e7430c2ac831f8e78fb5dfbda6ec8dcbd" : {
- "name" : "cool space babes",
- "type" : 2,
- "type_pretty" : "local file domain",
- "time_imported" : 1641204220
- }
- },
- "deleted" : {
- "6c6f63616c2066696c6573" : {
- "name" : "my files",
- "type" : 2,
- "type_pretty" : "local file domain",
- "time_deleted" : 1641204274,
- "time_imported" : 1641044491
- }
- }
- },
- "ipfs_multihashes" : {
- "55af93e0deabd08ce15ffb2b164b06d1254daab5a18d145e56fa98f71ddb6f11" : "QmReHtaET3dsgh7ho5NVyHb5U13UgJoGipSWbZsnuuM8tb"
- },
- "has_audio" : true,
- "num_frames" : 102,
- "num_words" : null,
- "is_inbox" : false,
- "is_local" : true,
- "is_trashed" : false,
- "is_deleted" : false,
- "has_exif" : false,
- "has_human_readable_embedded_metadata" : false,
- "has_icc_profile" : false,
- "known_urls" : [
- "https://gelbooru.com/index.php?page=post&s=view&id=4841557",
- "https://img2.gelbooru.com//images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg",
- "http://origin-orig.deviantart.net/ed31/f/2019/210/7/8/beachqueen_samus_by_dandonfuga-ddcu1xg.jpg"
- ],
- "tags" : {
- "6c6f63616c2074616773" : {
- "name" : "local tags",
- "type" : 5,
- "type_pretty" : "local tag service",
- "storage_tags" : {
- "0" : ["samus favourites"],
- "2" : ["process this later"]
- },
- "display_tags" : {
- "0" : ["samus favourites", "favourites"],
- "2" : ["process this later"]
- }
- },
- "37e3849bda234f53b0e9792a036d14d4f3a9a136d1cb939705dbcd5287941db4" : {
- "name" : "public tag repo",
- "type" : 1,
- "type_pretty" : "hydrus tag repository",
- "storage_tags" : {
- "0" : ["blonde_hair", "blue_eyes", "looking_at_viewer"],
- "1" : ["bodysuit"]
- },
- "display_tags" : {
- "0" : ["blonde hair", "blue_eyes", "looking at viewer"],
- "1" : ["bodysuit", "clothing"]
- }
- },
- "616c6c206b6e6f776e2074616773" : {
- "name" : "all known tags",
- "type" : 10,
- "type_pretty" : "virtual combined tag service",
- "storage_tags" : {
- "0" : ["samus favourites", "blonde_hair", "blue_eyes", "looking_at_viewer"],
- "1" : ["bodysuit"]
- },
- "display_tags" : {
- "0" : ["samus favourites", "favourites", "blonde hair", "blue_eyes", "looking at viewer"],
- "1" : ["bodysuit", "clothing"]
- }
- }
- }
- }
- ]
-}
-```
-```json title="And one where only_return_identifiers is true"
-{
- "metadata" : [
- {
- "file_id" : 123,
- "hash" : "4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2"
- },
- {
- "file_id" : 4567,
- "hash" : "3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82"
- }
- ]
-}
-```
-```json title="And where only_return_basic_information is true"
-{
- "metadata" : [
- {
- "file_id" : 123,
- "hash" : "4c77267f93415de0bc33b7725b8c331a809a924084bee03ab2f5fae1c6019eb2",
- "size" : 63405,
- "mime" : "image/jpeg",
- "ext" : ".jpg",
- "width" : 640,
- "height" : 480,
- "duration" : null,
- "has_audio" : false,
- "num_frames" : null,
- "num_words" : null,
- },
- {
- "file_id" : 4567,
- "hash" : "3e7cb9044fe81bda0d7a84b5cb781cba4e255e4871cba6ae8ecd8207850d5b82",
- "size" : 199713,
- "mime" : "video/webm",
- "ext" : ".webm",
- "width" : 1920,
- "height" : 1080,
- "duration" : 4040,
- "has_audio" : true,
- "num_frames" : 102,
- "num_words" : null,
- }
- ]
-}
-```
-
-#### basics
-
-Size is in bytes. Duration is in milliseconds, and may be an int or a float.
-
-`is_trashed` means if the file is currently in the trash but available on the hard disk. `is_deleted` means currently either in the trash or completely deleted from disk.
-
-`file_services` stores which file services the file is currently in and _deleted_ from. The entries are by the service key, same as for tags later on. In rare cases, the timestamps may be `null`, if they are unknown (e.g. a `time_deleted` for the file deleted before this information was tracked). The `time_modified` can also be null. Time modified is just the filesystem modified time for now, but it will evolve into more complicated storage in future with multiple locations (website post times) that'll be aggregated to a sensible value in UI.
-
-`ipfs_multihashes` stores the ipfs service key to any known multihash for the file.
-
-The `thumbnail_width` and `thumbnail_height` are a generally reliable prediction but aren't a promise. The actual thumbnail you get from [/get_files/thumbnail](#get_files_thumbnail) will be different if the user hasn't looked at it since changing their thumbnail options. You only get these rows for files that hydrus actually generates an actual thumbnail for. Things like pdf won't have it. You can use your own thumb, or ask the api and it'll give you a fixed fallback; those are mostly 200x200, but you can and should size them to whatever you want.
-
-#### tags
-
-The 'tags' structures are undergoing transition. Previously, this was a mess of different Objects in different domains, all `service_xxx_to_xxx_tags`, but they are being transitioned to the combined `tags` Object.
-
-`hide_service_keys_tags` is deprecated and will be deleted soon. When set to `false`, it shows the old `service_keys_to_statuses_to_tags` and `service_keys_to_statuses_to_display_tags` Objects.
-
-The `tags` structures are similar to the [/add_tags/add_tags](#add_tags_add_tags) scheme, excepting that the status numbers are:
-
-* 0 - current
-* 1 - pending
-* 2 - deleted
-* 3 - petitioned
-
-!!! note
- Since JSON Object keys must be strings, these status numbers are strings, not ints.
-
-To learn more about service names and keys on a client, use the [/get_services](#get_services) call.
-
-While the 'storage_tags' represent the actual tags stored on the database for a file, 'display_tags' reflect how tags appear in the UI, after siblings are collapsed and parents are added. If you want to edit a file's tags, refer to the storage tags. If you want to render to the user, use the display tags. The display tag calculation logic is very complicated; if the storage tags change, do not try to guess the new display tags yourself--just ask the API again.
-
-#### parameters
-
-If you ask with hashes rather than file_ids, hydrus will, by default, only return results when it has seen those hashes before. This is to stop the client making thousands of new file_id records in its database if you perform a scanning operation. If you ask about a hash the client has never encountered before--for which there is no file_id--you will get this style of result:
-
-```json title="Missing file_id example"
-{
- "metadata" : [
- {
- "file_id" : null,
- "hash" : "766da61f81323629f982bc1b71b5c1f9bba3f3ed61caf99906f7f26881c3ae93"
- }
- ]
-}
-```
-
-You can change this behaviour with `create_new_file_ids=true`, but bear in mind you will get a fairly 'empty' metadata result with lots of 'null' lines, so this is only useful for gathering the numerical ids for later Client API work.
-
-If you ask about any file_ids that do not exist, you'll get 404.
-
-If you set `only_return_basic_information=true`, this will be much faster for first-time requests than the full metadata result, but it will be slower for repeat requests. The full metadata object is cached after first fetch, the limited file info object is not.
-
-If you add `detailed_url_information=true`, a new entry, `detailed_known_urls`, will be added for each file, with a list of the same structure as /`add_urls/get_url_info`. This may be an expensive request if you are querying thousands of files at once.
-
-```json title="For example"
-{
- "detailed_known_urls": [
- {
- "normalised_url": "https://gelbooru.com/index.php?id=4841557&page=post&s=view",
- "url_type": 0,
- "url_type_string": "post url",
- "match_name": "gelbooru file page",
- "can_parse": true
- },
- {
- "normalised_url": "https://img2.gelbooru.com//images/80/c8/80c8646b4a49395fb36c805f316c49a9.jpg",
- "url_type": 5,
- "url_type_string": "unknown url",
- "match_name": "unknown url",
- "can_parse": false
- }
- ]
-}
-```
-
-
-### **GET `/get_files/file`** { id="get_files_file" }
-
-_Get a file._
-
-Restricted access:
-: YES. Search for Files permission needed. Additional search permission limits may apply.
-
-Required Headers: n/a
-
-Arguments :
-:
- * `file_id`: (numerical file id for the file)
- * `hash`: (a hexadecimal SHA256 hash for the file)
-
- Only use one. As with metadata fetching, you may only use the hash argument if you have access to all files. If you are tag-restricted, you will have to use a file_id in the last search you ran.
-
- ``` title="Example request"
- /get_files/file?file_id=452158
- ```
- ``` title="Example request"
- /get_files/file?hash=7f30c113810985b69014957c93bc25e8eb4cf3355dae36d8b9d011d8b0cf623a
- ```
-
-Response:
-: The file itself. You should get the correct mime type as the Content-Type header.
-
-
-### **GET `/get_files/thumbnail`** { id="get_files_thumbnail" }
-
-_Get a file's thumbnail._
-
-Restricted access:
-: YES. Search for Files permission needed. Additional search permission limits may apply.
-
-Required Headers: n/a
-
-Arguments:
-:
- * `file_id`: (numerical file id for the file)
- * `hash`: (a hexadecimal SHA256 hash for the file)
-
- Only use one. As with metadata fetching, you may only use the hash argument if you have access to all files. If you are tag-restricted, you will have to use a file_id in the last search you ran.
-
-``` title="Example request"
-/get_files/thumbnail?file_id=452158
-```
-``` title="Example request"
-/get_files/thumbnail?hash=7f30c113810985b69014957c93bc25e8eb4cf3355dae36d8b9d011d8b0cf623a
-```
-
-Response:
-: The thumbnail for the file. Some hydrus thumbs are jpegs, some are pngs. It should give you the correct image/jpeg or image/png Content-Type.
-
- If hydrus keeps no thumbnail for the filetype, for instance with pdfs, then you will get the same default 'pdf' icon you see in the client. If the file does not exist in the client, or the thumbnail was expected but is missing from storage, you will get the fallback 'hydrus' icon, again just as you would in the client itself. This request should never give a 404.
-
-!!! note
- If you get a 'default' filetype thumbnail like the pdf or hydrus one, you will be pulling the defaults straight from the hydrus/static folder. They will most likely be 200x200 pixels.
-
-
-
-## Managing File Relationships
-
-This refers to the File Relationships system, which includes 'potential duplicates', 'duplicates', and 'alternates'.
-
-This system is pending significant rework and expansion, so please do not get too married to some of the routines here. I am mostly just exposing my internal commands, so things are a little ugly/hacked. I expect duplicate and alternate groups to get some form of official identifier in future, which may end up being the way to refer and edit things here.
-
-Also, at least for now, 'Manage File Relationships' permission is not going to be bound by the search permission restrictions that normal file search does. Getting this permission allows you to search anything. I expect to add this permission filtering tech in future, particularly for file domains.
-
-_There is more work to do here, including adding various 'dissolve'/'undo' commands to break groups apart._
-
-### **GET `/manage_file_relationships/get_file_relationships`** { id="manage_file_relationships_get_file_relationships" }
-
-_Get the current relationships for one or more files._
-
-Restricted access:
-: YES. Manage File Relationships permission needed.
-
-Required Headers: n/a
-
-Arguments (in percent-encoded JSON):
-:
- * [files](#parameters_files)
- * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
-
-``` title="Example request"
-/manage_file_relationships/get_file_relationships?hash=ac940bb9026c430ea9530b4f4f6980a12d9432c2af8d9d39dfc67b05d91df11d
-```
-
-Response:
-: A JSON Object mapping the hashes to their relationships.
-``` json title="Example response"
-{
- "file_relationships" : {
- "ac940bb9026c430ea9530b4f4f6980a12d9432c2af8d9d39dfc67b05d91df11d" : {
- "is_king" : false,
- "king" : "8784afbfd8b59de3dcf2c13dc1be9d7cb0b3d376803c8a7a8b710c7c191bb657",
- "0" : [
- ],
- "1" : [],
- "3" : [
- "8bf267c4c021ae4fd7c4b90b0a381044539519f80d148359b0ce61ce1684fefe"
- ],
- "8" : [
- "8784afbfd8b59de3dcf2c13dc1be9d7cb0b3d376803c8a7a8b710c7c191bb657",
- "3fa8ef54811ec8c2d1892f4f08da01e7fc17eed863acae897eb30461b051d5c3"
- ]
- }
- }
-}
-```
-
-`is_king` and `king` relate to which file is the set best of a group. The king is usually the best representative of a group if you need to do comparisons between groups, and the 'get some pairs to filter'-style commands usually try to select the kings of the various to-be-compared duplicate groups.
-
-The relationships you get are filtered by the file domain. If you set the file domain to 'all known files', you will get every relationship a file has, including all deleted files, which is often less useful than you would think. The default, 'all my files' is usually most useful.
-
-**It is possible for the king to not be available, in which case `king` is null.** The king can be unavailable in several duplicate search contexts, generally when it is outside of the set file domain. For the default domain, 'all my files', the king will be available unless the user has deleted it. You have to deal with the king being unavailable--in this situation, your best bet is to just use the file itself as its own representative.
-
-A file that has no duplicates is considered to be in a duplicate group of size 1 and thus is always its own king.
-
-The numbers are from a duplicate status enum, as so:
-
-* 0 - potential duplicates
-* 1 - false positives
-* 3 - alternates
-* 8 - duplicates
-
-Note that because of JSON constraints, these are the string versions of the integers since they are Object keys.
-
-All the hashes given here are in 'all my files', i.e. not in the trash. A file may have duplicates that have long been deleted, but, like the null king above, they will not show here.
-
-### **GET `/manage_file_relationships/get_potentials_count`** { id="manage_file_relationships_get_potentials_count" }
-
-_Get the count of remaining potential duplicate pairs in a particular search domain. Exactly the same as the counts you see in the duplicate processing page._
-
-Restricted access:
-: YES. Manage File Relationships permission needed.
-
-Required Headers: n/a
-
-Arguments (in percent-encoded JSON):
-:
- * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
- * `tag_service_key_1`: (optional, default 'all known tags', a hex tag service key)
- * `tags_1`: (optional, default system:everything, a list of tags you wish to search for)
- * `tag_service_key_2`: (optional, default 'all known tags', a hex tag service key)
- * `tags_2`: (optional, default system:everything, a list of tags you wish to search for)
- * `potentials_search_type`: (optional, integer, default 0, regarding how the pairs should match the search(es))
- * `pixel_duplicates`: (optional, integer, default 1, regarding whether the pairs should be pixel duplicates)
- * `max_hamming_distance`: (optional, integer, default 4, the max 'search distance' of the pairs)
-
-``` title="Example request"
-/manage_file_relationships/get_potentials_count?tag_service_key_1=c1ba23c60cda1051349647a151321d43ef5894aacdfb4b4e333d6c4259d56c5f&tags_1=%5B%22dupes_to_process%22%2C%20%22system%3Awidth%3C400%22%5D&search_type=1&pixel_duplicates=2&max_hamming_distance=0&max_num_pairs=50
-```
-
-`tag_service_key` and `tags` work the same as [/get\_files/search\_files](#get_files_search_files). The `_2` variants are only useful if the `potentials_search_type` is 2. For now the file domain is locked to 'all my files'.
-
-`potentials_search_type` and `pixel_duplicates` are enums:
-
-* 0 - one file matches search 1
-* 1 - both files match search 1
-* 2 - one file matches search 1, the other 2
-
--and-
-
-* 0 - must be pixel duplicates
-* 1 - can be pixel duplicates
-* 2 - must not be pixel duplicates
-
-The `max_hamming_distance` is the same 'search distance' you see in the Client UI. A higher number means more speculative 'similar files' search. If `pixel_duplicates` is set to 'must be', then `max_hamming_distance` is obviously ignored.
-
-Response:
-: A JSON Object stating the count.
-``` json title="Example response"
-{
- "potential_duplicates_count" : 17
-}
-```
-
-If you confirm that a pair of potentials are duplicates, this may transitively collapse other potential pairs and decrease the count by more than 1.
-
-### **GET `/manage_file_relationships/get_potential_pairs`** { id="manage_file_relationships_get_potential_pairs" }
-
-_Get some potential duplicate pairs for a filtering workflow. Exactly the same as the 'duplicate filter' in the duplicate processing page._
-
-Restricted access:
-: YES. Manage File Relationships permission needed.
-
-Required Headers: n/a
-
-Arguments (in percent-encoded JSON):
-:
- * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
- * `tag_service_key_1`: (optional, default 'all known tags', a hex tag service key)
- * `tags_1`: (optional, default system:everything, a list of tags you wish to search for)
- * `tag_service_key_2`: (optional, default 'all known tags', a hex tag service key)
- * `tags_2`: (optional, default system:everything, a list of tags you wish to search for)
- * `potentials_search_type`: (optional, integer, default 0, regarding how the pairs should match the search(es))
- * `pixel_duplicates`: (optional, integer, default 1, regarding whether the pairs should be pixel duplicates)
- * `max_hamming_distance`: (optional, integer, default 4, the max 'search distance' of the pairs)
- * `max_num_pairs`: (optional, integer, defaults to client's option, how many pairs to get in a batch)
-
-``` title="Example request"
-/manage_file_relationships/get_potential_pairs?tag_service_key_1=c1ba23c60cda1051349647a151321d43ef5894aacdfb4b4e333d6c4259d56c5f&tags_1=%5B%22dupes_to_process%22%2C%20%22system%3Awidth%3C400%22%5D&search_type=1&pixel_duplicates=2&max_hamming_distance=0&max_num_pairs=50
-```
-
-The search arguments work the same as [/manage\_file\_relationships/get\_potentials\_count](#manage_file_relationships_get_potentials_count).
-
-`max_num_pairs` is simple and just caps how many pairs you get.
-
-Response:
-: A JSON Object listing a batch of hash pairs.
-```json title="Example response"
-{
- "potential_duplicate_pairs" : [
- [ "16470d6e73298cd75d9c7e8e2004810e047664679a660a9a3ba870b0fa3433d3", "7ed062dc76265d25abeee5425a859cfdf7ab26fd291f50b8de7ca381e04db079" ],
- [ "eeea390357f259b460219d9589b4fa11e326403208097b1a1fbe63653397b210", "9215dfd39667c273ddfae2b73d90106b11abd5fd3cbadcc2afefa526bb226608" ],
- [ "a1ea7d671245a3ae35932c603d4f3f85b0d0d40c5b70ffd78519e71945031788", "8e9592b2dfb436fe0a8e5fa15de26a34a6dfe4bca9d4363826fac367a9709b25" ]
- ]
-}
-```
-
-The selected pair sample and their order is strictly hardcoded for now (e.g. to guarantee that a decision will not invalidate any other pair in the batch, you shouldn't see the same file twice in a batch, nor two files in the same duplicate group). Treat it as the client filter does, where you fetch batches to process one after another. I expect to make it more flexible in future, in the client itself and here.
-
-You will see significantly fewer than `max_num_pairs` (and potential duplicate count) as you close to the last available pairs, and when there are none left, you will get an empty list.
-
-### **GET `/manage_file_relationships/get_random_potentials`** { id="manage_file_relationships_get_random_potentials" }
-
-_Get some random potentially duplicate file hashes. Exactly the same as the 'show some random potential dupes' button in the duplicate processing page._
-
-Restricted access:
-: YES. Manage File Relationships permission needed.
-
-Required Headers: n/a
-
-Arguments (in percent-encoded JSON):
-:
- * [file domain](#parameters_file_domain) (optional, defaults to 'all my files')
- * `tag_service_key_1`: (optional, default 'all known tags', a hex tag service key)
- * `tags_1`: (optional, default system:everything, a list of tags you wish to search for)
- * `tag_service_key_2`: (optional, default 'all known tags', a hex tag service key)
- * `tags_2`: (optional, default system:everything, a list of tags you wish to search for)
- * `potentials_search_type`: (optional, integer, default 0, regarding how the files should match the search(es))
- * `pixel_duplicates`: (optional, integer, default 1, regarding whether the files should be pixel duplicates)
- * `max_hamming_distance`: (optional, integer, default 4, the max 'search distance' of the files)
-
-``` title="Example request"
-/manage_file_relationships/get_random_potentials?tag_service_key_1=c1ba23c60cda1051349647a151321d43ef5894aacdfb4b4e333d6c4259d56c5f&tags_1=%5B%22dupes_to_process%22%2C%20%22system%3Awidth%3C400%22%5D&search_type=1&pixel_duplicates=2&max_hamming_distance=0
-```
-
-The arguments work the same as [/manage\_file\_relationships/get\_potentials\_count](#manage_file_relationships_get_potentials_count), with the caveat that `potentials_search_type` has special logic:
-
-* 0 - first file matches search 1
-* 1 - all files match search 1
-* 2 - first file matches search 1, the others 2
-
-Essentially, the first hash is the 'master' to which the others are paired. The other files will include every matching file.
-
-Response:
-: A JSON Object listing a group of hashes exactly as the client would.
-```json title="Example response"
-{
- "random_potential_duplicate_hashes" : [
- "16470d6e73298cd75d9c7e8e2004810e047664679a660a9a3ba870b0fa3433d3",
- "7ed062dc76265d25abeee5425a859cfdf7ab26fd291f50b8de7ca381e04db079",
- "9e0d6b928b726562d70e1f14a7b506ba987c6f9b7f2d2e723809bb11494c73e6",
- "9e01744819b5ff2a84dda321e3f1a326f40d0e7f037408ded9f18a11ee2b2da8"
- ]
-}
-```
-
-If there are no potential duplicate groups in the search, this returns an empty list.
-
-### **POST `/manage_file_relationships/set_file_relationships`** { id="manage_file_relationships_set_kings" }
-
-Set the relationships to the specified file pairs.
-
-Restricted access:
-: YES. Manage File Relationships permission needed.
-
-Required Headers:
-:
- * `Content-Type`: application/json
-
-Arguments (in JSON):
-:
- * `relationships`: (a list of Objects, one for each file-pair being set)
-
-Each Object is:
-
- * `hash_a`: (a hexadecimal SHA256 hash)
- * `hash_b`: (a hexadecimal SHA256 hash)
- * `relationship`: (integer enum for the relationship being set)
- * `do_default_content_merge`: (bool)
- * `delete_a`: (optional, bool, default false)
- * `delete_b`: (optional, bool, default false)
-
-`hash_a` and `hash_b` are normal hex SHA256 hashes for your file pair.
-
-`relationship` is one of this enum:
-
-* 0 - set as potential duplicates
-* 1 - set as false positives
-* 2 - set as same quality
-* 3 - set as alternates
-* 4 - set A as better
-* 7 - set B as better
-
-2, 4, and 7 all make the files 'duplicates' (8 under `get_file_relationships`), which, specifically, merges the two files' duplicate groups. 'same quality' has different duplicate content merge options to the better/worse choices, but it ultimately sets A>B. You obviously don't have to use 'B is better' if you prefer just to swap the hashes. Do what works for you.
-
-`do_default_content_merge` sets whether the user's duplicate content merge options should be loaded and applied to the files along with the relationship. Most operations in the client do this automatically, so the user may expect it to apply, but if you want to do content merge yourself, set this to false.
-
-`delete_a` and `delete_b` are booleans that select whether to delete A and/or B in the same operation as setting the relationship. You can also do this externally if you prefer.
-
-```json title="Example request body"
-{
- "relationships" : [
- {
- "hash_a" : "b54d09218e0d6efc964b78b070620a1fa19c7e069672b4c6313cee2c9b0623f2",
- "hash_b" : "bbaa9876dab238dcf5799bfd8319ed0bab805e844f45cf0de33f40697b11a845",
- "relationship" : 4,
- "do_default_content_merge" : true,
- "delete_b" : true
- },
- {
- "hash_a" : "22667427eaa221e2bd7ef405e1d2983846c863d40b2999ce8d1bf5f0c18f5fb2",
- "hash_b" : "65d228adfa722f3cd0363853a191898abe8bf92d9a514c6c7f3c89cfed0bf423",
- "relationship" : 4,
- "do_default_content_merge" : true,
- "delete_b" : true
- },
- {
- "hash_a" : "0480513ffec391b77ad8c4e57fe80e5b710adfa3cb6af19b02a0bd7920f2d3ec",
- "hash_b" : "5fab162576617b5c3fc8caabea53ce3ab1a3c8e0a16c16ae7b4e4a21eab168a7",
- "relationship" : 2,
- "do_default_content_merge" : true
- }
- ]
-}
-```
-
-Response:
-: 200 with no content.
-
-If you try to add an invalid or redundant relationship, for instance setting files that are already duplicates as potential duplicates, no changes are made.
-
-This is the file relationships request that is probably most likely to change in future. I may implement content merge options. I may move from file pairs to group identifiers. When I expand alternates, those file groups are going to support more variables.
-
-### **POST `/manage_file_relationships/set_kings`** { id="manage_file_relationships_set_kings" }
-
-Set the specified files to be the kings of their duplicate groups.
-
-Restricted access:
-: YES. Manage File Relationships permission needed.
-
-Required Headers:
-:
- * `Content-Type`: application/json
-
-Arguments (in JSON):
-:
- * [files](#parameters_files)
-
-```json title="Example request body"
-{
- "file_id" : 123
-}
-```
-
-Response:
-: 200 with no content.
-
-The files will be promoted to be the kings of their respective duplicate groups. If the file is already the king (also true for any file with no duplicates), this is idempotent. It also processes the files in the given order, so if you specify two files in the same group, the latter will be the king at the end of the request.
-
-## Managing Cookies and HTTP Headers
-
-This refers to the cookies held in the client's session manager, which are sent with network requests to different domains.
-
-### **GET `/manage_cookies/get_cookies`** { id="manage_cookies_get_cookies" }
-
-_Get the cookies for a particular domain._
-
-Restricted access:
-: YES. Manage Cookies permission needed.
-
-Required Headers: n/a
-
-Arguments:
-: * `domain`
-
- ``` title="Example request (for gelbooru.com)"
- /manage_cookies/get_cookies?domain=gelbooru.com
- ```
-
-
-Response:
-: A JSON Object listing all the cookies for that domain in \[ name, value, domain, path, expires \] format.
-```json title="Example response"
-{
- "cookies" : [
- ["__cfduid", "f1bef65041e54e93110a883360bc7e71", ".gelbooru.com", "/", 1596223327],
- ["pass_hash", "0b0833b797f108e340b315bc5463c324", "gelbooru.com", "/", 1585855361],
- ["user_id", "123456", "gelbooru.com", "/", 1585855361]
- ]
-}
-```
-
- Note that these variables are all strings except 'expires', which is either an integer timestamp or _null_ for session cookies.
-
- This request will also return any cookies for subdomains. The session system in hydrus generally stores cookies according to the second-level domain, so if you request for specific.someoverbooru.net, you will still get the cookies for someoverbooru.net and all its subdomains.
-
-### **POST `/manage_cookies/set_cookies`** { id="manage_cookies_set_cookies" }
-
-Set some new cookies for the client. This makes it easier to 'copy' a login from a web browser or similar to hydrus if hydrus's login system can't handle the site yet.
-
-Restricted access:
-: YES. Manage Cookies permission needed.
-
-Required Headers:
-:
- * `Content-Type`: application/json
-
-Arguments (in JSON):
-:
- * `cookies`: (a list of cookie rows in the same format as the GET request above)
-
-```json title="Example request body"
-{
- "cookies" : [
- ["PHPSESSID", "07669eb2a1a6e840e498bb6e0799f3fb", ".somesite.com", "/", 1627327719],
- ["tag_filter", "1", ".somesite.com", "/", 1627327719]
- ]
-}
-```
-
-You can set 'value' to be null, which will clear any existing cookie with the corresponding name, domain, and path (acting essentially as a delete).
-
-Expires can be null, but session cookies will time-out in hydrus after 60 minutes of non-use.
-
-### **POST `/manage_headers/set_user_agent`** { id="manage_headers_set_user_agent" }
-
-This sets the 'Global' User-Agent for the client, as typically editable under _network->data->manage http headers_, for instance if you want hydrus to appear as a specific browser associated with some cookies.
-
-Restricted access:
-: YES. Manage Cookies permission needed.
-
-Required Headers:
-:
- * `Content-Type`: application/json
-
-Arguments (in JSON):
-:
- * `user-agent`: (a string)
-
-```json title="Example request body"
-{
- "user-agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0"
-}
-```
-
-Send an empty string to reset the client back to the default User-Agent, which should be `Mozilla/5.0 (compatible; Hydrus Client)`.
-
-## Managing Pages
-
-This refers to the pages of the main client UI.
-
-### **GET `/manage_pages/get_pages`** { id="manage_pages_get_pages" }
-
-_Get the page structure of the current UI session._
-
-Restricted access:
-: YES. Manage Pages permission needed.
-
-Required Headers: n/a
-
-Arguments: n/a
-
-
-Response:
-: A JSON Object of the top-level page 'notebook' (page of pages) detailing its basic information and current sub-pages. Page of pages beneath it will list their own sub-page lists.
-```json title="Example response"
-{
- "pages" : {
- "name" : "top pages notebook",
- "page_key" : "3b28d8a59ec61834325eb6275d9df012860a1ecfd9e1246423059bc47fb6d5bd",
- "page_type" : 10,
- "selected" : true,
- "pages" : [
- {
- "name" : "files",
- "page_key" : "d436ff5109215199913705eb9a7669d8a6b67c52e41c3b42904db083255ca84d",
- "page_type" : 6,
- "selected" : false
- },
- {
- "name" : "thread watcher",
- "page_key" : "40887fa327edca01e1d69b533dddba4681b2c43e0b4ebee0576177852e8c32e7",
- "page_type" : 9,
- "selected" : false
- },
- {
- "name" : "pages",
- "page_key" : "2ee7fa4058e1e23f2bd9e915cdf9347ae90902a8622d6559ba019a83a785c4dc",
- "page_type" : 10,
- "selected" : true,
- "pages" : [
- {
- "name" : "urls",
- "page_key" : "9fe22cb760d9ee6de32575ed9f27b76b4c215179cf843d3f9044efeeca98411f",
- "page_type" : 7,
- "selected" : true
- },
- {
- "name" : "files",
- "page_key" : "2977d57fc9c588be783727bcd54225d577b44e8aa2f91e365a3eb3c3f580dc4e",
- "page_type" : 6,
- "selected" : false
- }
- ]
- }
- ]
- }
-}
-```
-
- The page types are as follows:
-
- * 1 - Gallery downloader
- * 2 - Simple downloader
- * 3 - Hard drive import
- * 5 - Petitions (used by repository janitors)
- * 6 - File search
- * 7 - URL downloader
- * 8 - Duplicates
- * 9 - Thread watcher
- * 10 - Page of pages
-
- The top page of pages will always be there, and always selected. 'selected' means which page is currently in view and will propagate down other page of pages until it terminates. It may terminate in an empty page of pages, so do not assume it will end on a 'media' page.
-
- The 'page_key' is a unique identifier for the page. It will stay the same for a particular page throughout the session, but new ones are generated on a client restart or other session reload.
-
-### **GET `/manage_pages/get_page_info`** { id="manage_pages_get_page_info" }
-
-_Get information about a specific page._
-
-!!! warning "Under Construction"
- This is under construction. The current call dumps a ton of info for different downloader pages. Please experiment in IRL situations and give feedback for now! I will flesh out this help with more enumeration info and examples as this gets nailed down. POST commands to alter pages (adding, removing, highlighting), will come later.
-
-Restricted access:
-: YES. Manage Pages permission needed.
-
-Required Headers: n/a
-
-Arguments:
-:
- * `page_key`: (hexadecimal page\_key as stated in [/manage\_pages/get\_pages](#manage_pages_get_pages))
- * `simple`: true or false (optional, defaulting to true)
-
- ``` title="Example request"
- /manage_pages/get_page_info?page_key=aebbf4b594e6986bddf1eeb0b5846a1e6bc4e07088e517aff166f1aeb1c3c9da&simple=true
- ```
-
-Response description
-: A JSON Object of the page's information. At present, this mostly means downloader information.
-```json title="Example response with simple = true"
-{
- "page_info" : {
- "name" : "threads",
- "page_key" : "aebbf4b594e6986bddf1eeb0b5846a1e6bc4e07088e517aff166f1aeb1c3c9da",
- "page_type" : 3,
- "management" : {
- "multiple_watcher_import" : {
- "watcher_imports" : [
- {
- "url" : "https://someimageboard.net/m/123456",
- "watcher_key" : "cf8c3525c57a46b0e5c2625812964364a2e801f8c49841c216b8f8d7a4d06d85",
- "created" : 1566164269,
- "last_check_time" : 1566164272,
- "next_check_time" : 1566174272,
- "files_paused" : false,
- "checking_paused" : false,
- "checking_status" : 0,
- "subject" : "gundam pictures",
- "imports" : {
- "status" : "4 successful (2 already in db)",
- "simple_status" : "4",
- "total_processed" : 4,
- "total_to_process" : 4
- },
- "gallery_log" : {
- "status" : "1 successful",
- "simple_status" : "1",
- "total_processed" : 1,
- "total_to_process" : 1
- }
- },
- {
- "url" : "https://someimageboard.net/a/1234",
- "watcher_key" : "6bc17555b76da5bde2dcceedc382cf7d23281aee6477c41b643cd144ec168510",
- "created" : 1566063125,
- "last_check_time" : 1566063133,
- "next_check_time" : 1566104272,
- "files_paused" : false,
- "checking_paused" : true,
- "checking_status" : 1,
- "subject" : "anime pictures",
- "imports" : {
- "status" : "124 successful (22 already in db), 2 previously deleted",
- "simple_status" : "124",
- "total_processed" : 124,
- "total_to_process" : 124
- },
- "gallery_log" : {
- "status" : "3 successful",
- "simple_status" : "3",
- "total_processed" : 3,
- "total_to_process" : 3
- }
- }
- ]
- },
- "highlight" : "cf8c3525c57a46b0e5c2625812964364a2e801f8c49841c216b8f8d7a4d06d85"
- }
- },
- "media" : {
- "num_files" : 4
- }
-}
-```
-
- As you can see, even the 'simple' mode can get very large. Imagine that response for a page watching 100 threads! Turning simple mode off will display every import item, gallery log entry, and all hashes in the media (thumbnail) panel.
-
- For this first version, the five importer pages--hdd import, simple downloader, url downloader, gallery page, and watcher page--all give rich info based on their specific variables. The first three only have one importer/gallery log combo, but the latter two of course can have multiple. The "imports" and "gallery_log" entries are all in the same data format.
-
-
-### **POST `/manage_pages/add_files`** { id="manage_pages_add_files" }
-
-_Add files to a page._
-
-Restricted access:
-: YES. Manage Pages permission needed.
-
-Required Headers:
-:
- * `Content-Type`: application/json
-
-Arguments (in JSON):
-:
- * `page_key`: (the page key for the page you wish to add files to)
- * [files](#parameters_files)
-
-The files you set will be appended to the given page, just like a thumbnail drag and drop operation. The page key is the same as fetched in the [/manage\_pages/get\_pages](#manage_pages_get_pages) call.
-
-```json title="Example request body"
-{
- "page_key" : "af98318b6eece15fef3cf0378385ce759bfe056916f6e12157cd928eb56c1f18",
- "file_ids" : [123, 124, 125]
-}
-```
-
-Response:
-: 200 with no content. If the page key is not found, this will 404.
-
-### **POST `/manage_pages/focus_page`** { id="manage_pages_focus_page" }
-
-_'Show' a page in the main GUI, making it the current page in view. If it is already the current page, no change is made._
-
-Restricted access:
-: YES. Manage Pages permission needed.
-
-Required Headers:
-:
- * `Content-Type`: application/json
-
-Arguments (in JSON):
-:
- * `page_key`: (the page key for the page you wish to show)
-
-The page key is the same as fetched in the [/manage\_pages/get\_pages](#manage_pages_get_pages) call.
-
-```json title="Example request body"
-{
- "page_key" : "af98318b6eece15fef3cf0378385ce759bfe056916f6e12157cd928eb56c1f18"
-}
-```
-
-Response:
-: 200 with no content. If the page key is not found, this will 404.
-
-
-### **POST `/manage_pages/refresh_page`** { id="manage_pages_refresh_page" }
-
-_Refresh a page in the main GUI. Like hitting F5 in the client, this obviously makes file search pages perform their search again, but for other page types it will force the currently in-view files to be re-sorted._
-
-Restricted access:
-: YES. Manage Pages permission needed.
-
-Required Headers:
-:
- * `Content-Type`: application/json
-
-Arguments (in JSON):
-:
- * `page_key`: (the page key for the page you wish to refresh)
-
-The page key is the same as fetched in the [/manage\_pages/get\_pages](#manage_pages_get_pages) call. If a file search page is not set to 'searching immediately', a 'refresh' command does nothing.
-
-```json title="Example request body"
-{
- "page_key" : "af98318b6eece15fef3cf0378385ce759bfe056916f6e12157cd928eb56c1f18"
-}
-```
-
-Response:
-: 200 with no content. If the page key is not found, this will 404.
-
-## Managing the Database
-
-### **POST `/manage_database/lock_on`** { id="manage_database_lock_on" }
-
-_Pause the client's database activity and disconnect the current connection._
-
-Restricted access:
-: YES. Manage Database permission needed.
-
-Arguments: None
-
-This is a hacky prototype. It commands the client database to pause its job queue and release its connection (and related file locks and journal files). This puts the client in a similar position as a long VACUUM command--it'll hang in there, but not much will work, and since the UI async code isn't great yet, the UI may lock up after a minute or two. If you would like to automate database backup without shutting the client down, this is the thing to play with.
-
-This should return pretty quick, but it will wait up to five seconds for the database to actually disconnect. If there is a big job (like a VACUUM) current going on, it may take substantially longer to finish that up and process this STOP command. You might like to check for the existence of a journal file in the db dir just to be safe.
-
-As long as this lock is on, all Client API calls except the unlock command will return 503. (This is a decent way to test the current lock status, too)
-
-### **POST `/manage_database/lock_off`** { id="manage_database_lock_off" }
-
-_Reconnect the client's database and resume activity._
-
-Restricted access:
-: YES. Manage Database permission needed.
-
-Arguments: None
-
-This is the obvious complement to the lock. The client will resume processing its job queue and will catch up. If the UI was frozen, it should free up in a few seconds, just like after a big VACUUM.
-
-### **GET `/manage_database/mr_bones`** { id="manage_database_mr_bones" }
-
-_Get the data from help->how boned am I?. This is a simple Object of numbers just for hacky advanced purposes if you want to build up some stats in the background. The numbers are the same as the dialog shows, so double check that to confirm what means what._
-
-Restricted access:
-: YES. Manage Database permission needed.
-
-Arguments: None
-
-```json title="Example response"
-{
- "boned_stats" : {
- "num_inbox" : 8356,
- "num_archive" : 229,
- "num_deleted" : 7010,
- "size_inbox" : 7052596762,
- "size_archive" : 262911007,
- "size_deleted" : 13742290193,
- "earliest_import_time" : 1451408539,
- "total_viewtime" : [3280, 41621, 2932, 83021],
- "total_alternate_files" : 265,
- "total_duplicate_files" : 125,
- "total_potential_pairs" : 3252
- }
-}
-```
diff --git a/docs/getting_started_importing.md b/docs/getting_started_importing.md
index ca82a76d..fedee9c9 100644
--- a/docs/getting_started_importing.md
+++ b/docs/getting_started_importing.md
@@ -19,7 +19,7 @@ Drag-and-drop one or more folders or files into Hydrus.
This will open the `import files` window. Here you can add files or folders, or delete files from the import queue. Let Hydrus parse what it will update and then look over the options. By default the option to delete original files after succesful import (if it's ignored for any reason or already present in Hydrus for example) is not checked, activate on your own risk. In `file import options` you can find some settings for minimum and maximum file size, resolution, and whether to import previously deleted files or not.
From here there's two options: `import now` which will just import as is, and `add tags before import >>` which lets you set up some rules to add tags to files on import.
-Examples are keeping filename as a tag, add folders as tag (useful if you have some sort of folder based organisation scheme), or load tags from an accompanying .txt file generated by some other program.
+Examples are keeping filename as a tag, add folders as tag (useful if you have some sort of folder based organisation scheme), or load tags from an [accompanying text file](advanced_sidecars.md) generated by some other program.
Once you're done click apply (or `import now`) and Hydrus will start processing the files. Exact duplicates are not imported so if you had dupes spread out you will end up with only one file in the end. If files *look* similar but Hydrus imports both then that's a job for the [dupe filter](duplicates.md) as there is some difference even if you can't tell it by eye. A common one is compression giving files with different file sizes, but otherwise looking identical or files with extra meta data baked into them.
@@ -41,7 +41,7 @@ If you use a drag and drop to open a file inside an image editing program, remem
You can also copy the files by right-clicking and going down `share -> copy -> files` and then pasting the files where you want them.
### Export
-You can also export files with tags, either in filename or as a side-car .txt file by right-clicking and going down `share -> export -> files`. Have a look at the settings and then press `export`.
+You can also export files with tags, either in filename or as a [sidecar file](advanced_sidecars.md) by right-clicking and going down `share -> export -> files`. Have a look at the settings and then press `export`.
You can create folders to export files into by using backslashes on Windows (`\`) and slashes on Linux (`/`) in the filename. This can be combined with the patterns listed in the pattern shortcut button dropdown. As example `[series]\{filehash}` will export files into folders named after the `series:` namespaced tags on the files, all files tagged with one series goes into one folder, files tagged with another series goes into another folder as seen in the image below.
![](images/export_files.png)
@@ -56,12 +56,12 @@ Under `file -> import and export folders` you'll find options for setting up aut
### Import folders
![](images/import_folder.png)
-To import tags you have to add a tag service under the `filename tagging` section.
+Like with a manual import, if you wish you can import tags by parsing filenames or [loading sidecars](advanced_sidecars.md).
### Export folders
![](images/export_folder.png)
-You can currently not export tags in a .txt file with export folder liks you can when doing normal exports unfortunately.
+Like with manual export, you can set the filenames using a tag pattern, and you can [export to sidecars](advanced_sidecars.md) too.
## Importing and exporting tags
While you can import and export tags together with images sometimes you just don't want to deal with the files.
diff --git a/docs/images/sidecars_example_json_export.png b/docs/images/sidecars_example_json_export.png
new file mode 100644
index 00000000..81aaea71
Binary files /dev/null and b/docs/images/sidecars_example_json_export.png differ
diff --git a/docs/images/sidecars_example_json_import.png b/docs/images/sidecars_example_json_import.png
new file mode 100644
index 00000000..3df2b788
Binary files /dev/null and b/docs/images/sidecars_example_json_import.png differ
diff --git a/docs/images/sidecars_example_manual_export.png b/docs/images/sidecars_example_manual_export.png
new file mode 100644
index 00000000..3a5dacd4
Binary files /dev/null and b/docs/images/sidecars_example_manual_export.png differ
diff --git a/docs/images/sidecars_example_manual_import.png b/docs/images/sidecars_example_manual_import.png
new file mode 100644
index 00000000..7be19cac
Binary files /dev/null and b/docs/images/sidecars_example_manual_import.png differ
diff --git a/docs/old_changelog.html b/docs/old_changelog.html
index fa034c36..65052283 100644
--- a/docs/old_changelog.html
+++ b/docs/old_changelog.html
@@ -34,6 +34,81 @@
+ -
+
+
+ downloaders
+ - twitter took down the API we were using, breaking all our nice twitter downloaders! argh!
+ - a user has figured out a basic new downloader that grabs the tweets amongst the first twenty tweets-and-retweets of an account. yes, only the first twenty max, and usually fewer. because this is a big change, the client will ask about it when you update. if you have some complicated situation where you are working on the old default twitter downloaders and don't want them deleted, you can select 'no' on the dialog it throws up, but everyone else wants to say 'yes'. then check your twitter subs: make sure they moved to the new downloader, and you probably want to make them check more frequently too.
+ - given the rate of changes at twitter, I think we can expect more changes and blocks in future. I don't know whether nitter will be viable alternative, so if the artists you like end up on a nice simple booru _anywhere_, I strongly recommend just moving there. twitter appears to be explicitly moving to non-third-party-friendly
+ - thanks to a user's work, the 'danbooru - get webm ugoira' parser is fixed!
+ - thanks to a user's work, the deviant art parser is updated to get the highest res image in more situations!
+ - thanks to a user's work, the pixiv downloader now gets the artist note, in japanese (and translated, if there is one), and a 'medium:ai generated' tag!
+ sidecars
+ - I wrote some sidecar help here! https://hydrusnetwork.github.io/hydrus/advanced_sidecars.html
+ - when the client parses files for import, the 'does this look like a sidecar?' test now also checks that the base component of the base filename (e.g. 'Image123' from 'Image123.jpg.txt') actually appears in the list of non-txt/json/xml ext files. a random yo.txt file out of nowhere will now be inspected in case it is secretly a jpeg again, for good or ill
+ - when you drop some files on the client, the number of files skipped because they looked like sidecars is now stated in the status label
+ - fixed a typo bug that meant tags imported from sidecars were not being properly cleaned, despite preview appearance otherwise, for instance ':)', which in hydrus needs to be secretly stored as '::)' was being imported as ')'
+ - as a special case, tags that in hydrus are secretly '::)' will be converted to ':)' on export to sidecar too, the inverse of the above problem. there may be some other tag cleaning quirks to undo here, so let me know what you run into
+ related tags overhaul
+ - the 'related tags' suggestion system, turned on under _options->tag suggestions_, has several changes, including some prototype tech I'd love feedback on
+ - first off, there are two new search buttons, 'new 1' and 'new 2' ('2' is available on repositories only).. these use an upgraded statistical search and scoring system that a user worked on and sent in. I have butchered his specific namespace searching system to something more general/flexible and easy for me to maintain, but it works better and more comprehensibly than my old method! give it a go and let me know how each button does--the first one will be fast but less useful on the PTR, the second will be slower but generally give richer results (although it cannot do tags with too-high count)
+ - the new search routine works on multiple files, so 'related tags' now shows on tag dialogs launched from a selection of thumbnails!
+ - also, all the related search buttons now search any selection of tags you make!!! so if you can't remember that character's name, just click on the series or another character they are often with and hit the search, and you should get a whole bunch appear
+ - I am going to keep working on this in the future. the new buttons will become the only buttons, I'll try and mitigate the prototype search limitations, add some cancel tech, move to a time-based search length like the current buttons, and I'll add more settings, including for filtering so we aren't looking up related tags for 'page:x' and so on. I'm interested in knowing how you get on with IRL data. are there too many recommendations (is the tolerance too high?)? is the sorting good (is the stuff at the top relevant or often just noise?)?
+ misc
+ - all users can now copy their service keys (which are a technical non-changing hex identifier for your client's services) from the review services window--advanced mode is no longer needed. this may be useful as the client api transitions to service keys
+ - when a job in the downloader search log generates new jobs (e.g. fetches the next page), the new job(s) are now inserted after the parent. previously, they were appended to the end of the list. this changes how ngugs operate, converting their searches from interleaved to sequential!
+ - restarting search log jobs now also places the new job after the restarted job
+ - when you create a new export folder, if you have default metadata export sidecar settings from a previous manual file export, the program now asks if you want those for the new export folder or an empty list. previously, it just assigned the saved default, which could be jarring if it was saved from ages ago
+ - added a migration guide to the running from source help. also brushed up some language and fixed a bunch of borked title weights in that document
+ - the max initial and periodic file limits in subscriptions is now 50k when in advanced mode. I can't promise that would be nice though!
+ - the file history chart no longer says that inbox and delete time tracking are new
+ misc fixes
+ - fixed a cursor type detection test that was stopping the cursor from hiding immediately when you do a media viewer drag in Qt6
+ - fixed an issue where 'clear deletion record' calls were not deleting from the newer 'all my files' domain. the erroneous extra records will be searched for and scrubbed on update
+ - fixed the issue where if you had the new 'unnamespaced input gives (any namespace) wildcard results' search option on, you couldn't add any novel tags in WRITE autocomplete contexts like 'manage tags'!!! it could only offer the automatically converted wildcard tags as suggested input, which of course aren't appropriate for a WRITE context. the way I ultimately fixed this was horrible; the whole thing needs more work to deal with clever logic like this better, so let me know if you get any more trouble here
+ - I think I fixed an infinite hang when trying to add certain siblings in manage tag siblings. I believe this was occuring when the dialog was testing if the new pair would create a loop when the sibling structure already contains a loop. now it throws up a message and breaks the test
+ - fixed an issue where certain system:filetype predicates would spawn apparent duplicates of themselves instead of removing on double-click. images+audio+video+swf+pdf was one example. it was a 'all the image types' vs 'list of (all the) image types' conversion/comparison/sorting issue
+ client api
+ - **this is later than I expected, but as was planned last year, I am clearing up several obsolete parameters and data structures this week. mostly it is bad service name-identification that seemed simple or flexible to support but just added maintenance debt, induced bad implementation practises, and hindered future expansions. if you have a custom api script, please read on--and if you have not yet moved to the alternatives, do so before updating!**
+ - **all `...service_name...` parameters are officially obsolete! they will still work via some legacy hacks, so old scripts shouldn't break, but they are no longer documented. please move to the `...service_key...` alternates as soon as reasonably possible (check out `/get_services` if you need to learn about service keys)**
+ - **`/add_tags/get_tag_services` is removed! use `/get_services` instead!**
+ - **`hide_service_names_tags`, previously made default true, is removed and its data structures `service_names_to_statuses_to_...` are also gone! move to the new `tags` structure.**
+ - **`hide_service_keys_tags` is now default true. it will be removed in 4 weeks or so. same deal as with `service_names_to_statuses_to_...`--move to `tags`**
+ - **`system_inbox` and `system_archive` are removed from `/get_files/search_files`! just use 'system:inbox/archive' in the tags list**
+ - **the 'set_file_relationships' command from last week has been reworked to have a nicer Object parameter with a new name. please check the updated help!** normally I wouldn't change something so quick, but we are still in early prototype, so I'm ok shifting it (and the old method still works lmao, but I'll clear that code out in a few weeks, so please move over--the Object will be much nicer to expand in future, which I forgot about in v513)
+ many Client API commands now support modern file domain objects, meaning you can search a UNION of file services and 'deleted-from' file services. affected commands are
+ - * /add_files/delete_files
+ - * /add_files/undelete_files
+ - * /add_tags/search_tags
+ - * /get_files/search_files
+ - * /manage_file_relationships/get_everything
+ - a new `/get_service` call now lets you ask about an individual service by service name or service key, basically a parameterised /get_services
+ - the `/manage_pages/get_pages` and `/manage_pages/get_page_info` calls now give the `page_state`, a new enum that says if the page is ready, initialised, searching, or search-cancelled
+ - to reduce duplicate argument spam, the client api help now specifies the complicated 'these files' and now 'this file domain' arguments into sub-sections, and the commands that use them just point to the subsections. check it out--it makes sense when you look at it.
+ - `/add_tags/add_tags` now raises 400 if you give an invalid content action (e.g. pending to a local tag service). previously it skipped these rows silently
+ - added and updated unit tests and help for the above changes
+ - client api version is now 41
+ boring optimisation
+ - when you are looking at a search log or file log, if entries are added, removed, or moved around, all the log entries that have changed row # now update (previously it just sent a redraw signal for the new rows, not the second-order affected rows that were shuffled up/down. many access routines for these logs are sped up
+ - file log status checking is completely rewritten. the ways it searches, caches and optimises the 'which is the next item with x status' queues is faster and requires far less maintenance. large import queues have less overhead, so the in and outs of general download work should scale up much better now
+ - the main data cache that stores rendered images, image tiles, and thumbnails now maintains itself far more efficiently. there was a hellish O(n) overhead when adding or removing an item which has been reduced to constant time. this gonk was being spammed every few minutes during normal memory maintenance, when hundreds of thumbs can be purged at once. clients with tens of thousands of thumbnails in memory will maintain that list far more smoothly
+ - physical file delete is now more efficient, requiring far fewer hard drive hits to delete a media file. it is also far less aggressive, with a new setting in _options->files and trash_ that sets how long to wait between individual file deletes, default 250ms. before, it was full LFG mode with minor delays every hundred/thousand jobs, and since it takes a write lock, it was lagging out thumbnail load when hitting a lot of work. the daemon here also shuts down faster if caught working during program shut down
+ boring code cleanup
+ - refactored some parsing routines to be more flexible
+ - added some more dictionary and enum type testing to the client api parameter parsing routines. error messages should be better!
+ - improved how `/add_tags/add_tags` parsing works. ensuring both access methods check all types and report nicer errors
+ - cleaned up the `/search_files/file_metadata` call's parsing, moving to the new generalised method and smoothing out some old code flow. it now checks hashes against the last search, too
+ - cleaned up `/manage_pages/add_files` similarly
+ - cleaned up how tag services are parsed and their errors reported in the client api
+ - the client api is better about processing the file identifiers you give it in the same order you gave
+ - fixed bad 'potentials_search_type'/'search_type' inconsistency in the client api help examples
+ - obviously a bunch of client api unit test and help cleanup to account for the obsolete stuff and various other changes here
+ - updated a bunch of the client api unit tests to handle some of the new parsing
+ - fixed the remaining 'randomly fail due to complex counting logic' potential count unit tests. turns out there were like seven more of them
+
+
-
diff --git a/docs/running_from_source.md b/docs/running_from_source.md
index 11ccc472..6d3fb49e 100644
--- a/docs/running_from_source.md
+++ b/docs/running_from_source.md
@@ -186,7 +186,7 @@ The first start will take a little longer. It will operate just like a normal bu
If you want to redirect your database or use any other launch arguments, then copy 'client.command' to 'client-user.command' and edit it, inserting your desired db path. Run this instead of 'client.command'. New `git pull` commands will not affect 'client-user.command'.
-## Simple Updating Guide
+### Simple Updating Guide
To update, you do the same thing as for the extract builds.
@@ -195,11 +195,33 @@ To update, you do the same thing as for the extract builds.
If you get a library version error when you try to boot, run the venv setup again. It is worth doing this anyway, every now and then, just to stay up to date.
-## doing it manually { id="what_you_need" }
+### Migrating from an Existing Install
+
+Many users start out using one of the official built releases and decide to move to source. There is lots of information [here](database_migration.md) about how to migrate the database, but for your purposes, the simple method is this:
+
+**If you never moved your database to another place and do not use -d/--db_dir launch parameter**
+
+1. Follow the above guide to get the source install working in a new folder on a fresh database
+2. **MAKE A BACKUP OF EVERYTHING**
+3. Delete everything from the source install's `db` directory.
+4. Move your built release's entire `db` directory to the source.
+5. Run your source release again--it should load your old db no problem!
+6. Update your backup routine to point to the new source install location.
+
+**If you moved your database to another location and use the -d/--db_dir launch parameter**
+
+1. Follow the above guide to get the source install working in a new folder on a fresh database (without -db_dir)
+2. **MAKE A BACKUP OF EVERYTHING**
+3. Just to be neat, delete the .db files, .log files, and client_files folder from the source install's `db` directory.
+4. Run the source install with --db_dir just as you would the built executable--it should load your old db no problem!
+
+## Doing it Yourself { id="what_you_need" }
_This is for advanced users only._
-Inside the extract should be client.py and server.py. You will be treating these basically the same as the 'client' and 'server' executables--you should be able to launch them the same way and they take the same launch parameters as the exes.
+_If you have never used python before, do not try this. If the easy setup scripts failed for you and you don't know what happened, please contact hydev before trying this, as the thing that went wrong there will probably go much more wrong here._
+
+You can also set up the environment yourself. Inside the extract should be client.py and server.py. You will be treating these basically the same as the 'client' and 'server' executables--with the right environment, you should be able to launch them the same way and they take the same launch parameters as the exes.
Hydrus needs a whole bunch of libraries, so let's now set your python up. I **strongly** recommend you create a virtual environment. It is easy and doesn't mess up your system python.
@@ -210,14 +232,14 @@ To create a new venv environment:
* Open a terminal at your hydrus extract folder. If `python3` doesn't work, use `python`.
* `python3 -m pip install virtualenv` (if you need it)
* `python3 -m venv venv`
-* `source venv/bin/activate`
+* `source venv/bin/activate` (`CALL venv\Scripts\activate.bat` in Windows cmd)
* `python -m pip install --upgrade pip`
* `python -m pip install --upgrade wheel`
!!! info "venvs"
- That `source venv/bin/activate` line turns on your venv. You should see your terminal note you are now in it. A venv is an isolated environment of python that you can install modules to without worrying about breaking something system-wide. **Ideally, you do not want to install python modules to your system python.**
+ That `source venv/bin/activate` line turns on your venv. You should see your terminal prompt note you are now in it. A venv is an isolated environment of python that you can install modules to without worrying about breaking something system-wide. **Ideally, you do not want to install python modules to your system python.**
- This activate line will be needed every time you alter your venv or run the `client.py`/`server.py` files. You can easily tuck this venv activation line into a launch script--check the easy setup files for examples.
+ This activate line will be needed every time you alter your venv or run the `client.py`/`server.py` files. You can easily tuck this into a launch script--check the easy setup files for examples.
On Windows Powershell, the command is `.\venv\Scripts\activate`, but you may find the whole deal is done much easier in cmd than Powershell. When in Powershell, just type `cmd` to get an old fashioned command line. In cmd, the launch command is just `venv\scripts\activate.bat`, no leading period.
@@ -227,9 +249,9 @@ To create a new venv environment:
python -m pip install -r requirements.txt
```
-If you need different versions of libraries, check the cut-up requirements.txts the 'advanced' easy-setup uses in `install_dir/static/requirements/advanced`. Check and compare their contents to the main requirements.txt to see what is going on.
+If you need different versions of libraries, check the cut-up requirements.txts the 'advanced' easy-setup uses in `install_dir/static/requirements/advanced`. Check and compare their contents to the main requirements.txt to see what is going on. You'll likely need the newer OpenCV on Python 3.10, for instance.
-## Qt { id="qt" }
+### Qt { id="qt" }
Qt is the UI library. You can run PySide2, PySide6, PyQt5, or PyQt6. A wrapper library called `qtpy` allows this. The default is PySide6, but if it is missing, qtpy will fall back to an available alternative. For PyQt5 or PyQt6, you need an extra Chart module, so go:
@@ -245,17 +267,15 @@ If you want to set QT_API in a batch file, do this:
`set QT_API=pyqt6`
-If you run Windows 8.1 or Ubuntu 18.04, you cannot run Qt6. Please try PySide2 or PyQt5.
+If you run <= Windows 8.1 or Ubuntu 18.04, you cannot run Qt6. Try PySide2 or PyQt5.
-## mpv support { id="mpv" }
+### mpv { id="mpv" }
MPV is optional and complicated, but it is great, so it is worth the time to figure out!
-As well as the python wrapper, 'python-mpv' (which is in the requirements.txt), you also need the underlying library. This is _not_ mpv the program, but 'libmpv', often called 'libmpv1'.
+As well as the python wrapper, 'python-mpv' (which is in the requirements.txt), you also need the underlying dev library. This is _not_ mpv the program, but 'libmpv', often called 'libmpv1'.
-For Windows, the dll builds are [here](https://sourceforge.net/projects/mpv-player-windows/files/libmpv/), although getting a stable version can be difficult. Just put it in your hydrus base install directory. Check the links in the easy-setup guide above for good versions.
-
-You can also just grab the 'mpv-1.dll'/'mpv-2.dll' I bundle in my extractable Windows release.
+For Windows, the dll builds are [here](https://sourceforge.net/projects/mpv-player-windows/files/libmpv/), although getting a stable version can be difficult. Just put it in your hydrus base install directory. Check the links in the easy-setup guide above for good versions. You can also just grab the 'mpv-1.dll'/'mpv-2.dll' I bundle in my extractable Windows release.
If you are on Linux, you can usually get 'libmpv1' like so:
@@ -265,13 +285,11 @@ On macOS, you should be able to get it with `brew install mpv`, but you are like
Hit _help->about_ to see your mpv status. If you don't have it, it will present an error popup box with more info.
-## SQLite { id="sqlite" }
+### SQLite { id="sqlite" }
-If you can, update python's SQLite--it'll improve performance.
+If you can, update python's SQLite--it'll improve performance. The SQLite that comes with stock python is usually quite old, so you'll get a significant boost in speed. In some python deployments, the built-in SQLite not compiled with neat features like Fast Text Search (FTS) that hydrus needs.
-On Windows, get the 64-bit sqlite3.dll [here](https://www.sqlite.org/download.html), and just drop it in your base install directory.
-
-You can also just grab the 'sqlite3.dll' I bundle in my extractable Windows release.
+On Windows, get the 64-bit sqlite3.dll [here](https://www.sqlite.org/download.html), and just drop it in your base install directory. You can also just grab the 'sqlite3.dll' I bundle in my extractable Windows release.
You _may_ be able to update your SQLite on Linux or macOS with:
@@ -279,27 +297,27 @@ You _may_ be able to update your SQLite on Linux or macOS with:
* (activate your venv)
* `python -m pip install pysqlite3`
-But it isn't a big deal.
+But as long as the program launches, it usually isn't a big deal.
!!! warning "Extremely safe no way it can go wrong"
If you want to update sqlite for your system python install, you can also drop it into `C:\Python38\DLLs` or wherever you have python installed. You'll be overwriting the old file, so make a backup if you want to (I have never had trouble updating like this, however).
+
+ A user who made a Windows venv with Anaconda reported they had to replace the sqlite3.dll in their conda env at `~/.conda/envs//Library/bin/sqlite3.dll`.
-## FFMPEG { id="ffmpeg" }
+### FFMPEG { id="ffmpeg" }
-If you don't have FFMPEG in your PATH and you want to import anything more fun than jpegs, you will need to put a static [FFMPEG](https://ffmpeg.org/) executable in your PATH or the `install_dir/bin` directory. [This](https://www.gyan.dev/ffmpeg/builds/ffmpeg-release-full.7z) should always point to a new build.
+If you don't have FFMPEG in your PATH and you want to import anything more fun than jpegs, you will need to put a static [FFMPEG](https://ffmpeg.org/) executable in your PATH or the `install_dir/bin` directory. [This](https://www.gyan.dev/ffmpeg/builds/ffmpeg-release-full.7z) should always point to a new build for Windows. Alternately, you can just copy the exe from one of my extractable Windows releases.
-Alternately, you can just copy the exe from one of my extractable Windows releases.
+### Running It { id="running_it" }
-## running it { id="running_it" }
-
-Once you have everything set up, client.py and server.py should look for and run off client.db and server.db just like the executables. You are looking at entering something like this into the terminal:
+Once you have everything set up, client.py and server.py should look for and run off client.db and server.db just like the executables. You can use the 'client.bat/sh/command' scripts in the install dir or use them as inspiration for your own. In any case, you are looking at entering something like this into the terminal:
```
source venv/bin/activate
python client.py
```
-This will look in the 'db' directory by default, but you can use the [launch arguments](launch_arguments.md) just like for the executables. For example, this could be your client-user.sh file:
+This will use the 'db' directory for your database by default, but you can use the [launch arguments](launch_arguments.md) just like for the executables. For example, this could be your client-user.sh file:
```
#!/bin/bash
@@ -308,13 +326,11 @@ source venv/bin/activate
python client.py -d="/path/to/database"
```
-I develop hydrus on and am most experienced with Windows, so the program is more stable and reasonable on that. I do not have as much experience with Linux or macOS, but I still appreciate and will work on your Linux/macOS bug reports.
+### Building these Docs
-## Building the docs
+When running from source you may want to [build the hydrus help docs](about_docs.md) yourself. You can also check the `setup_help` scripts in the install directory.
-When running from source you may want to [build the hydrus help docs](about_docs.md) yourself.
-
-## building packages on windows { id="windows_build" }
+### Building Packages on Windows { id="windows_build" }
Almost everything you get through pip is provided as pre-compiled 'wheels' these days, but if you get an error about Visual Studio C++ when you try to pip something, you have two choices:
@@ -335,13 +351,15 @@ Trust me, just do this, it will save a ton of headaches!
_Update:_ On Windows 11, in 2023-01, I had trouble with the above. There's a couple '11' SDKs that installed ok, but the vcbuildtools stuff had unusual errors. I hadn't done this in years, so maybe they are broken for Windows 10 too! The good news is that a basic stock Win 11 install with Python 3.10 is fine getting everything on our requirements and even making a build without any extra compiler tech.
-## additional windows info { id="additional_windows" }
+### Additional Windows Info { id="additional_windows" }
This does not matter much any more, but in the old days, building modules like lz4 and lxml was a complete nightmare, and hooking up Visual Studio was even more difficult. [This page](http://www.lfd.uci.edu/~gohlke/pythonlibs/) has a lot of prebuilt binaries--I have found it very helpful many times.
I have a fair bit of experience with Windows python, so send me a mail if you need help.
-## my code { id="my_code" }
+## My Code { id="my_code" }
+
+I develop hydrus on and am most experienced with Windows, so the program is more stable and reasonable on that. I do not have as much experience with Linux or macOS, but I still appreciate and will work on your Linux/macOS bug reports.
My coding style is unusual and unprofessional. Everything is pretty much hacked together. If you are interested in how things work, please do look through the source and ask me if you don't understand something.
diff --git a/hydrus/client/ClientCaches.py b/hydrus/client/ClientCaches.py
index 240a5ee5..9f453b0c 100644
--- a/hydrus/client/ClientCaches.py
+++ b/hydrus/client/ClientCaches.py
@@ -42,15 +42,17 @@ class DataCache( object ):
if key not in self._keys_to_data:
return
-
+
+
+ ( data, size_estimate ) = self._keys_to_data[ key ]
del self._keys_to_data[ key ]
- self._RecalcMemoryUsage()
+ self._total_estimated_memory_footprint -= size_estimate
if HG.cache_report_mode:
- HydrusData.ShowText( 'Cache "{}" removing "{}". Current size {}.'.format( self._name, key, HydrusData.ConvertValueRangeToBytes( self._total_estimated_memory_footprint, self._cache_size ) ) )
+ HydrusData.ShowText( 'Cache "{}" removing "{}", size "{}". Current size {}.'.format( self._name, key, HydrusData.ToHumanBytes( size_estimate ), HydrusData.ConvertValueRangeToBytes( self._total_estimated_memory_footprint, self._cache_size ) ) )
@@ -61,9 +63,27 @@ class DataCache( object ):
self._Delete( deletee_key )
- def _RecalcMemoryUsage( self ):
+ def _GetData( self, key ):
- self._total_estimated_memory_footprint = sum( ( data.GetEstimatedMemoryFootprint() for data in self._keys_to_data.values() ) )
+ if key not in self._keys_to_data:
+
+ raise Exception( 'Cache error! Looking for {}, but it was missing.'.format( key ) )
+
+
+ self._TouchKey( key )
+
+ ( data, size_estimate ) = self._keys_to_data[ key ]
+
+ new_estimate = data.GetEstimatedMemoryFootprint()
+
+ if new_estimate != size_estimate:
+
+ self._total_estimated_memory_footprint += new_estimate - size_estimate
+
+ self._keys_to_data[ key ] = ( data, new_estimate )
+
+
+ return data
def _TouchKey( self, key ):
@@ -99,19 +119,21 @@ class DataCache( object ):
self._DeleteItem()
- self._keys_to_data[ key ] = data
+ size_estimate = data.GetEstimatedMemoryFootprint()
+
+ self._keys_to_data[ key ] = ( data, size_estimate )
+
+ self._total_estimated_memory_footprint += size_estimate
self._TouchKey( key )
- self._RecalcMemoryUsage()
-
if HG.cache_report_mode:
HydrusData.ShowText(
'Cache "{}" adding "{}" ({}). Current size {}.'.format(
self._name,
key,
- HydrusData.ToHumanBytes( data.GetEstimatedMemoryFootprint() ),
+ HydrusData.ToHumanBytes( size_estimate ),
HydrusData.ConvertValueRangeToBytes( self._total_estimated_memory_footprint, self._cache_size )
)
)
@@ -132,14 +154,7 @@ class DataCache( object ):
with self._lock:
- if key not in self._keys_to_data:
-
- raise Exception( 'Cache error! Looking for {}, but it was missing.'.format( key ) )
-
-
- self._TouchKey( key )
-
- return self._keys_to_data[ key ]
+ return self._GetData( key )
@@ -149,9 +164,7 @@ class DataCache( object ):
if key in self._keys_to_data:
- self._TouchKey( key )
-
- return self._keys_to_data[ key ]
+ return self._GetData( key )
else:
@@ -180,6 +193,11 @@ class DataCache( object ):
with self._lock:
+ while self._total_estimated_memory_footprint > self._cache_size:
+
+ self._DeleteItem()
+
+
while True:
if len( self._keys_fifo ) == 0:
diff --git a/hydrus/client/ClientConstants.py b/hydrus/client/ClientConstants.py
index 0ab82859..4c1358d5 100644
--- a/hydrus/client/ClientConstants.py
+++ b/hydrus/client/ClientConstants.py
@@ -303,6 +303,11 @@ page_file_count_display_string_lookup = {
PAGE_FILE_COUNT_DISPLAY_NONE : 'for no pages'
}
+PAGE_STATE_NORMAL = 0
+PAGE_STATE_INITIALISING = 1
+PAGE_STATE_SEARCHING = 2
+PAGE_STATE_SEARCHING_CANCELLED = 3
+
SHUTDOWN_TIMESTAMP_VACUUM = 0
SHUTDOWN_TIMESTAMP_FATTEN_AC_CACHE = 1
SHUTDOWN_TIMESTAMP_DELETE_ORPHANS = 2
diff --git a/hydrus/client/ClientFiles.py b/hydrus/client/ClientFiles.py
index 096f8106..3f3c02b7 100644
--- a/hydrus/client/ClientFiles.py
+++ b/hydrus/client/ClientFiles.py
@@ -51,8 +51,8 @@ regen_file_enum_to_str_lookup = {
REGENERATE_FILE_DATA_JOB_REFIT_THUMBNAIL : 'regenerate thumbnail if incorrect size',
REGENERATE_FILE_DATA_JOB_OTHER_HASHES : 'regenerate non-standard hashes',
REGENERATE_FILE_DATA_JOB_DELETE_NEIGHBOUR_DUPES : 'delete duplicate neighbours with incorrect file extension',
- REGENERATE_FILE_DATA_JOB_FILE_INTEGRITY_PRESENCE_REMOVE_RECORD : 'if file is missing, remove record (no delete record)',
- REGENERATE_FILE_DATA_JOB_FILE_INTEGRITY_PRESENCE_DELETE_RECORD : 'if file is missing, remove record (leave delete record)',
+ REGENERATE_FILE_DATA_JOB_FILE_INTEGRITY_PRESENCE_REMOVE_RECORD : 'if file is missing, remove record (leave no delete record)',
+ REGENERATE_FILE_DATA_JOB_FILE_INTEGRITY_PRESENCE_DELETE_RECORD : 'if file is missing, remove record (leave a delete record)',
REGENERATE_FILE_DATA_JOB_FILE_INTEGRITY_PRESENCE_TRY_URL : 'if file is missing, then if has URL try to redownload',
REGENERATE_FILE_DATA_JOB_FILE_INTEGRITY_PRESENCE_TRY_URL_ELSE_REMOVE_RECORD : 'if file is missing, then if has URL try to redownload, else remove record',
REGENERATE_FILE_DATA_JOB_FILE_INTEGRITY_PRESENCE_LOG_ONLY : 'if file is missing, note it in log',
@@ -185,6 +185,8 @@ def GetAllFilePaths( raw_paths, do_human_sort = True, clear_out_sidecars = True
file_paths = []
+ num_sidecars = 0
+
paths_to_process = list( raw_paths )
while len( paths_to_process ) > 0:
@@ -230,24 +232,49 @@ def GetAllFilePaths( raw_paths, do_human_sort = True, clear_out_sidecars = True
HydrusData.HumanTextSort( file_paths )
+ num_files_with_sidecars = len( file_paths )
+
if clear_out_sidecars:
exts = [ '.txt', '.json', '.xml' ]
- def not_a_sidecar( p ):
+ def has_sidecar_ext( p ):
if True in ( p.endswith( ext ) for ext in exts ):
- return False
+ return True
- return True
+ return False
- file_paths = [ path for path in file_paths if not_a_sidecar( path ) ]
+ def get_base_prefix_component( p ):
+
+ base_prefix = os.path.basename( p )
+
+ if '.' in base_prefix:
+
+ base_prefix = base_prefix.split( '.', 1 )[0]
+
+
+ return base_prefix
+
+
+ # let's get all the 'Image123' in our 'path/to/Image123.jpg' list
+ all_non_ext_prefix_components = { get_base_prefix_component( file_path ) for file_path in file_paths if not has_sidecar_ext( file_path ) }
+
+ def looks_like_a_sidecar( p ):
+
+ # if we have Image123.txt, that's probably a sidecar!
+ return has_sidecar_ext( p ) and get_base_prefix_component( p ) in all_non_ext_prefix_components
+
+
+ file_paths = [ path for path in file_paths if not looks_like_a_sidecar( path ) ]
- return file_paths
+ num_sidecars = num_files_with_sidecars - len( file_paths )
+
+ return ( file_paths, num_sidecars )
class ClientFilesManager( object ):
@@ -259,7 +286,7 @@ class ClientFilesManager( object ):
self._prefixes_to_locations = {}
- self._new_physical_file_deletes = threading.Event()
+ self._physical_file_delete_wait = threading.Event()
self._locations_to_free_space = {}
@@ -1172,11 +1199,11 @@ class ClientFilesManager( object ):
def DoDeferredPhysicalDeletes( self ):
+ wait_period = self._controller.new_options.GetInteger( 'ms_to_wait_between_physical_file_deletes' ) / 1000
+
num_files_deleted = 0
num_thumbnails_deleted = 0
- pauser = HydrusData.BigJobPauser()
-
while not HG.started_shutdown:
with self._rwlock.write:
@@ -1190,9 +1217,18 @@ class ClientFilesManager( object ):
if file_hash is not None:
+ media_result = self._controller.Read( 'media_result', file_hash )
+
+ expected_mime = media_result.GetMime()
+
try:
- ( path, mime ) = self._LookForFilePath( file_hash )
+ path = self._GenerateExpectedFilePath( file_hash, expected_mime )
+
+ if not os.path.exists( path ):
+
+ ( path, actual_mime ) = self._LookForFilePath( file_hash )
+
ClientPaths.DeletePath( path )
@@ -1200,7 +1236,7 @@ class ClientFilesManager( object ):
except HydrusExceptions.FileMissingException:
- pass
+ HydrusData.Print( 'Wanted to physically delete the "{}" file, with expected mime "{}", but it was not found!'.format( file_hash.hex(), expected_mime ) )
@@ -1214,6 +1250,10 @@ class ClientFilesManager( object ):
num_thumbnails_deleted += 1
+ else:
+
+ HydrusData.Print( 'Wanted to physically delete the "{}" thumbnail, but it was not found!'.format( file_hash.hex() ) )
+
self._controller.WriteSynchronous( 'clear_deferred_physical_delete', file_hash = file_hash, thumbnail_hash = thumbnail_hash )
@@ -1224,7 +1264,9 @@ class ClientFilesManager( object ):
- pauser.Pause()
+ self._physical_file_delete_wait.wait( wait_period )
+
+ self._physical_file_delete_wait.clear()
if num_files_deleted > 0 or num_thumbnails_deleted > 0:
@@ -1335,11 +1377,6 @@ class ClientFilesManager( object ):
return os.path.exists( path )
- def NotifyNewPhysicalFileDeletes( self ):
-
- self._new_physical_file_deletes.set()
-
-
def Rebalance( self, job_key ):
try:
@@ -1502,7 +1539,7 @@ class ClientFilesManager( object ):
def shutdown( self ):
- self._new_physical_file_deletes.set()
+ self._physical_file_delete_wait.set()
class FilesMaintenanceManager( object ):
diff --git a/hydrus/client/ClientOptions.py b/hydrus/client/ClientOptions.py
index 7a2e2122..0f0f3474 100644
--- a/hydrus/client/ClientOptions.py
+++ b/hydrus/client/ClientOptions.py
@@ -482,6 +482,8 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
self._dictionary[ 'integers' ][ 'human_bytes_sig_figs' ] = 3
+ self._dictionary[ 'integers' ][ 'ms_to_wait_between_physical_file_deletes' ] = 250
+
#
self._dictionary[ 'keys' ] = {}
diff --git a/hydrus/client/ClientSearch.py b/hydrus/client/ClientSearch.py
index 52e6f5b0..2c69750b 100644
--- a/hydrus/client/ClientSearch.py
+++ b/hydrus/client/ClientSearch.py
@@ -1576,7 +1576,7 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
if predicate_type == PREDICATE_TYPE_SYSTEM_MIME and value is not None:
- value = tuple( ConvertSpecificFiletypesToSummary( value ) )
+ value = tuple( sorted( ConvertSpecificFiletypesToSummary( value ) ) )
if predicate_type == PREDICATE_TYPE_OR_CONTAINER:
@@ -1771,6 +1771,11 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
self._value = serialisable_value
+ if self._predicate_type == PREDICATE_TYPE_SYSTEM_MIME and self._value is not None:
+
+ self._value = tuple( sorted( ConvertSpecificFiletypesToSummary( self._value ) ) )
+
+
if isinstance( self._value, list ):
@@ -1863,7 +1868,7 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
summary_mimes = ConvertSpecificFiletypesToSummary( specific_mimes )
- serialisable_value = tuple( summary_mimes )
+ serialisable_value = tuple( sorted( summary_mimes ) )
new_serialisable_info = ( predicate_type, serialisable_value, inclusive )
@@ -3100,7 +3105,7 @@ class ParsedAutocompleteText( object ):
return 'AC Tag Text: {}'.format( self.raw_input )
- def _GetSearchText( self, always_autocompleting: bool, force_do_not_collapse: bool = False, allow_unnamespaced_search_gives_any_namespace_wildcards: bool = True ) -> str:
+ def _GetSearchText( self, always_autocompleting: bool, force_do_not_collapse: bool = False, allow_auto_wildcard_conversion: bool = False ) -> str:
text = CollapseWildcardCharacters( self.raw_content )
@@ -3114,7 +3119,7 @@ class ParsedAutocompleteText( object ):
text = ConvertTagToSearchable( text )
- if allow_unnamespaced_search_gives_any_namespace_wildcards and self._tag_autocomplete_options.UnnamespacedSearchGivesAnyNamespaceWildcards():
+ if allow_auto_wildcard_conversion and self._tag_autocomplete_options.UnnamespacedSearchGivesAnyNamespaceWildcards():
if ':' not in text:
@@ -3149,12 +3154,12 @@ class ParsedAutocompleteText( object ):
def GetAddTagPredicate( self ):
- return Predicate( PREDICATE_TYPE_TAG, self.raw_content )
+ return Predicate( PREDICATE_TYPE_TAG, self.raw_content, self.inclusive )
- def GetImmediateFileSearchPredicate( self ):
+ def GetImmediateFileSearchPredicate( self, allow_auto_wildcard_conversion ):
- non_tag_predicates = self.GetNonTagFileSearchPredicates()
+ non_tag_predicates = self.GetNonTagFileSearchPredicates( allow_auto_wildcard_conversion )
if len( non_tag_predicates ) > 0:
@@ -3166,7 +3171,7 @@ class ParsedAutocompleteText( object ):
return tag_search_predicate
- def GetNonTagFileSearchPredicates( self ):
+ def GetNonTagFileSearchPredicates( self, allow_auto_wildcard_conversion ):
predicates = []
@@ -3182,7 +3187,7 @@ class ParsedAutocompleteText( object ):
predicates.append( predicate )
- elif self.IsExplicitWildcard():
+ elif self.IsExplicitWildcard( allow_auto_wildcard_conversion ):
search_texts = []
@@ -3199,7 +3204,7 @@ class ParsedAutocompleteText( object ):
for always_autocompleting in always_autocompleting_values:
- search_texts.append( self._GetSearchText( always_autocompleting, allow_unnamespaced_search_gives_any_namespace_wildcards = allow_unnamespaced_search_gives_any_namespace_wildcards, force_do_not_collapse = True ) )
+ search_texts.append( self._GetSearchText( always_autocompleting, allow_auto_wildcard_conversion = allow_unnamespaced_search_gives_any_namespace_wildcards, force_do_not_collapse = True ) )
@@ -3220,9 +3225,9 @@ class ParsedAutocompleteText( object ):
return predicates
- def GetSearchText( self, always_autocompleting: bool ):
+ def GetSearchText( self, always_autocompleting: bool, allow_auto_wildcard_conversion = True ):
- return self._GetSearchText( always_autocompleting )
+ return self._GetSearchText( always_autocompleting, allow_auto_wildcard_conversion = allow_auto_wildcard_conversion )
def GetTagAutocompleteOptions( self ):
@@ -3232,7 +3237,7 @@ class ParsedAutocompleteText( object ):
def IsAcceptableForTagSearches( self ):
- search_text = self._GetSearchText( False )
+ search_text = self._GetSearchText( False, allow_auto_wildcard_conversion = True )
if search_text == '':
@@ -3267,7 +3272,7 @@ class ParsedAutocompleteText( object ):
def IsAcceptableForFileSearches( self ):
- search_text = self._GetSearchText( False )
+ search_text = self._GetSearchText( False, allow_auto_wildcard_conversion = True )
if len( search_text ) == 0:
@@ -3287,10 +3292,10 @@ class ParsedAutocompleteText( object ):
return self.raw_input == ''
- def IsExplicitWildcard( self ):
+ def IsExplicitWildcard( self, allow_auto_wildcard_conversion ):
# user has intentionally put a '*' in
- return '*' in self.raw_content or self._GetSearchText( False ).startswith( '*:' )
+ return '*' in self.raw_content or self._GetSearchText( False, allow_auto_wildcard_conversion = allow_auto_wildcard_conversion ).startswith( '*:' )
def IsNamespaceSearch( self ):
@@ -3300,9 +3305,9 @@ class ParsedAutocompleteText( object ):
return SearchTextIsNamespaceFetchAll( search_text ) or SearchTextIsNamespaceBareFetchAll( search_text )
- def IsTagSearch( self ):
+ def IsTagSearch( self, allow_auto_wildcard_conversion ):
- if self.IsEmpty() or self.IsExplicitWildcard() or self.IsNamespaceSearch():
+ if self.IsEmpty() or self.IsExplicitWildcard( allow_auto_wildcard_conversion ) or self.IsNamespaceSearch():
return False
diff --git a/hydrus/client/db/ClientDB.py b/hydrus/client/db/ClientDB.py
index 80b29a77..e4ead699 100644
--- a/hydrus/client/db/ClientDB.py
+++ b/hydrus/client/db/ClientDB.py
@@ -5158,6 +5158,246 @@ class DB( HydrusDB.HydrusDB ):
return predicates
+ def _GetRelatedTagsNewOneTag( self, tag_display_type, file_service_id, tag_service_id, search_tag_id ):
+
+ # a user provided the basic idea here
+
+ # we are saying get me all the tags for all the hashes this tag has
+ # specifying namespace is critical to increase search speed, otherwise we actually are searching all tags for tags
+ # we also call this with single specific file domains to keep things fast
+
+ # also this thing searches in fixed file domain to get fast
+
+ # this table selection is hacky as anything, but simpler than GetMappingAndTagTables for now
+
+ mappings_table_names = []
+
+ if file_service_id == self.modules_services.combined_file_service_id:
+
+ ( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = ClientDBMappingsStorage.GenerateMappingsTableNames( tag_service_id )
+
+ mappings_table_names.extend( [ current_mappings_table_name, pending_mappings_table_name ] )
+
+ else:
+
+ if tag_display_type == ClientTags.TAG_DISPLAY_ACTUAL:
+
+ ( cache_current_display_mappings_table_name, cache_pending_display_mappings_table_name ) = ClientDBMappingsStorage.GenerateSpecificDisplayMappingsCacheTableNames( file_service_id, tag_service_id )
+
+ mappings_table_names.extend( [ cache_current_display_mappings_table_name, cache_pending_display_mappings_table_name ] )
+
+ else:
+
+ statuses_to_table_names = self.modules_mappings_storage.GetFastestStorageMappingTableNames( file_service_id, tag_service_id )
+
+ mappings_table_names.extend( [ statuses_to_table_names[ HC.CONTENT_STATUS_CURRENT ], statuses_to_table_names[ HC.CONTENT_STATUS_PENDING ] ] )
+
+
+
+ results = collections.Counter()
+
+ tags_table_name = self.modules_tag_search.GetTagsTableName( file_service_id, tag_service_id )
+
+ # while this searches pending and current tags, it does not cross-reference current and pending on the same file, oh well!
+
+ for mappings_table_name in mappings_table_names:
+
+ search_predicate = 'hash_id IN ( SELECT hash_id FROM {} WHERE tag_id = {} )'.format( mappings_table_name, search_tag_id )
+
+ query = 'SELECT tag_id, COUNT( * ) FROM {} CROSS JOIN {} USING ( tag_id ) WHERE {} GROUP BY subtag_id;'.format( mappings_table_name, tags_table_name, search_predicate )
+
+ results.update( dict( self._Execute( query ).fetchall() ) )
+
+
+ return results
+
+
+ def _GetRelatedTagsNew( self, file_service_key, tag_service_key, search_tags, max_results = 100, concurrence_threshold = 0.04, total_search_tag_count_threshold = 500000 ):
+
+ # a user provided the basic idea here
+
+ if len( search_tags ) == 0:
+
+ return [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, value = 'no search tags to work with!' ) ]
+
+
+ tag_display_type = ClientTags.TAG_DISPLAY_ACTUAL
+
+ tag_service_id = self.modules_services.GetServiceId( tag_service_key )
+ file_service_id = self.modules_services.GetServiceId( file_service_key )
+
+ if tag_display_type == ClientTags.TAG_DISPLAY_ACTUAL:
+
+ search_tags = self.modules_tag_siblings.GetIdeals( tag_display_type, tag_service_key, search_tags )
+
+ # I had a thing here that added the parents, but it gave some whack results compared to what you expected
+
+
+ search_tag_ids_to_search_tags = self.modules_tags_local_cache.GetTagIdsToTags( tags = search_tags )
+
+ with self._MakeTemporaryIntegerTable( search_tag_ids_to_search_tags.keys(), 'tag_id' ) as temp_tag_id_table_name:
+
+ search_tag_ids_to_total_counts = collections.Counter( { tag_id : current_count + pending_count for ( tag_id, current_count, pending_count ) in self.modules_mappings_counts.GetCountsForTags( tag_display_type, file_service_id, tag_service_id, temp_tag_id_table_name ) } )
+
+
+ #
+
+ search_tags = set()
+
+ for ( search_tag_id, count ) in search_tag_ids_to_total_counts.items():
+
+ # pending only
+ if count == 0:
+
+ continue
+
+
+ search_tags.add( search_tag_ids_to_search_tags[ search_tag_id ] )
+
+
+ if len( search_tags ) == 0:
+
+ return [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, value = 'not enough data in search domain' ) ]
+
+
+ #
+
+ search_tag_ids_flat_sorted_ascending = sorted( search_tag_ids_to_total_counts.items(), key = lambda row: row[1] )
+
+ search_tags = set()
+ total_count = 0
+
+ # TODO: I think I would rather rework this into a time threshold thing like the old related tags stuff.
+ # so, should ditch the culling and instead make all the requests cancellable. just keep searching until we are out of time, then put results together
+
+ # TODO: Another option as I vacillate on 'all my files' vs 'all known files' would be to incorporate that into the search timer
+ # do all my files first, then replace that with all known files results until we run out of time (only do this for repositories)
+
+ # we don't really want to use '1girl' and friends as search tags here, since the search domain is so huge
+ # so, we go for the smallest count tags first. they have interesting suggestions
+ # searching all known files is gonkmode, so we curtail our max search size
+
+ if file_service_key == CC.COMBINED_FILE_SERVICE_KEY:
+
+ total_search_tag_count_threshold /= 25
+
+
+ for ( search_tag_id, count ) in search_tag_ids_flat_sorted_ascending:
+
+ # we don't want the total domain to be too large either. death by a thousand cuts
+ if total_count + count > total_search_tag_count_threshold:
+
+ break
+
+
+ total_count += count
+
+ search_tags.add( search_tag_ids_to_search_tags[ search_tag_id ] )
+
+
+ if len( search_tags ) == 0:
+
+ return [ ClientSearch.Predicate( ClientSearch.PREDICATE_TYPE_TAG, value = 'search domain too big' ) ]
+
+
+ #
+
+ search_tag_ids_to_tag_ids_to_matching_counts = {}
+
+ for search_tag in search_tags:
+
+ search_tag_id = self.modules_tags_local_cache.GetTagId( search_tag )
+
+ tag_ids_to_matching_counts = self._GetRelatedTagsNewOneTag( tag_display_type, file_service_id, tag_service_id, search_tag_id )
+
+ if search_tag_id in tag_ids_to_matching_counts:
+
+ del tag_ids_to_matching_counts[ search_tag_id ] # duh, don't recommend your 100% matching self
+
+
+ search_tag_ids_to_tag_ids_to_matching_counts[ search_tag_id ] = tag_ids_to_matching_counts
+
+
+ #
+
+ # ok we have a bunch of counts here for different search tags, so let's figure out some normalised scores and merge them all
+ #
+ # the master score is: number matching mappings found / square_root( suggestion_tag_count * search_tag_count )
+ #
+ # I don't really know what this *is*, but the user did it and it seems to make nice scores, so hooray
+ # the dude said it was arbitrary and could do with tuning, so we'll see how it goes
+
+ all_tag_ids = set()
+
+ for tag_ids_to_matching_counts in search_tag_ids_to_tag_ids_to_matching_counts.values():
+
+ all_tag_ids.update( tag_ids_to_matching_counts.keys() )
+
+
+ all_tag_ids.difference_update( search_tag_ids_to_search_tags.keys() )
+
+ with self._MakeTemporaryIntegerTable( all_tag_ids, 'tag_id' ) as temp_tag_id_table_name:
+
+ tag_ids_to_total_counts = { tag_id : current_count + pending_count for ( tag_id, current_count, pending_count ) in self.modules_mappings_counts.GetCountsForTags( tag_display_type, file_service_id, tag_service_id, temp_tag_id_table_name ) }
+
+
+ tag_ids_to_total_counts.update( search_tag_ids_to_total_counts )
+
+ tag_ids_to_scores = collections.Counter()
+
+ for ( search_tag_id, tag_ids_to_matching_counts ) in search_tag_ids_to_tag_ids_to_matching_counts.items():
+
+ if search_tag_id not in tag_ids_to_total_counts:
+
+ continue
+
+
+ search_tag_count = tag_ids_to_total_counts[ search_tag_id ]
+
+ search_tag_is_unnamespaced = HydrusTags.IsUnnamespaced( search_tag_ids_to_search_tags[ search_tag_id ] )
+
+ for ( tag_id, matching_count ) in tag_ids_to_matching_counts.items():
+
+ if matching_count / search_tag_count < concurrence_threshold:
+
+ continue
+
+
+ if tag_id not in tag_ids_to_total_counts:
+
+ continue
+
+
+ suggestion_tag_count = tag_ids_to_total_counts[ tag_id ]
+
+ score = matching_count / ( ( suggestion_tag_count * search_tag_count ) ** 0.5 )
+
+ # sophisticated hydev score-tuning
+ if search_tag_is_unnamespaced:
+
+ score /= 3
+
+
+ tag_ids_to_scores[ tag_id ] += score
+
+
+
+ results_flat_sorted_descending = sorted( tag_ids_to_scores.items(), key = lambda row: row[1], reverse = True )
+
+ tag_ids_to_scores = dict( results_flat_sorted_descending[ : max_results ] )
+
+ #
+
+ inclusive = True
+ pending_count = 0
+
+ tag_ids_to_full_counts = { tag_id : ( int( score * 1000 ), None, pending_count, None ) for ( tag_id, score ) in tag_ids_to_scores.items() }
+
+ predicates = self.modules_tag_display.GeneratePredicatesFromTagIdsAndCounts( tag_display_type, tag_service_id, tag_ids_to_full_counts, inclusive )
+
+ return predicates
+
+
def _GetRepositoryThumbnailHashesIDoNotHave( self, service_key ):
service_id = self.modules_services.GetServiceId( service_key )
@@ -7379,6 +7619,7 @@ class DB( HydrusDB.HydrusDB ):
elif action == 'services': result = self.modules_services.GetServices( *args, **kwargs )
elif action == 'similar_files_maintenance_status': result = self.modules_similar_files.GetMaintenanceStatus( *args, **kwargs )
elif action == 'related_tags': result = self._GetRelatedTags( *args, **kwargs )
+ elif action == 'related_tags_new': result = self._GetRelatedTagsNew( *args, **kwargs )
elif action == 'tag_display_application': result = self.modules_tag_display.GetApplication( *args, **kwargs )
elif action == 'tag_display_maintenance_status': result = self._CacheTagDisplayGetApplicationStatusNumbers( *args, **kwargs )
elif action == 'tag_parents': result = self.modules_tag_parents.GetTagParents( *args, **kwargs )
@@ -10762,6 +11003,116 @@ class DB( HydrusDB.HydrusDB ):
+ if version == 513:
+
+ try:
+
+ self._controller.frame_splash_status.SetSubtext( 'cleaning some surplus records' )
+
+ # clear deletion record wasn't purging on 'all my files'
+
+ all_my_deleted_hash_ids = set( self.modules_files_storage.GetDeletedHashIdsList( self.modules_services.combined_local_media_service_id ) )
+
+ all_local_current_hash_ids = self.modules_files_storage.GetCurrentHashIdsList( self.modules_services.combined_local_file_service_id )
+ all_local_deleted_hash_ids = self.modules_files_storage.GetDeletedHashIdsList( self.modules_services.combined_local_file_service_id )
+
+ erroneous_hash_ids = all_my_deleted_hash_ids.difference( all_local_current_hash_ids ).difference( all_local_deleted_hash_ids )
+
+ if len( erroneous_hash_ids ) > 0:
+
+ service_ids_to_nums_cleared = self.modules_files_storage.ClearLocalDeleteRecord( erroneous_hash_ids )
+
+ self._ExecuteMany( 'UPDATE service_info SET info = info + ? WHERE service_id = ? AND info_type = ?;', ( ( -num_cleared, clear_service_id, HC.SERVICE_INFO_NUM_DELETED_FILES ) for ( clear_service_id, num_cleared ) in service_ids_to_nums_cleared.items() ) )
+
+
+ except:
+
+ HydrusData.PrintException( e )
+
+ message = 'Trying to clean up some bad delete records failed! Please let hydrus dev know!'
+
+ self.pub_initial_message( message )
+
+
+ try:
+
+ def ask_what_to_do_twitter_stuff():
+
+ message = 'Twitter removed their old API that we were using, breaking all the old downloaders! I am going to delete your old twitter downloaders and add a new limited one that can only get the first ~20 tweets of a profile. Make sure to check your subscriptions are linked to it, and you might want to speed up their check times! OK?'
+
+ from hydrus.client.gui import ClientGUIDialogsQuick
+
+ result = ClientGUIDialogsQuick.GetYesNo( None, message, title = 'Swap to new twitter downloader?', yes_label = 'do it', no_label = 'do not do it, I need to keep the old broken stuff' )
+
+ return result == QW.QDialog.Accepted
+
+
+ do_twitter_stuff = self._controller.CallBlockingToQt( None, ask_what_to_do_twitter_stuff )
+
+ domain_manager = self.modules_serialisable.GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER )
+
+ domain_manager.Initialise()
+
+ #
+
+ domain_manager.OverwriteDefaultParsers( [
+ 'danbooru file page parser - get webm ugoira',
+ 'deviant art file page parser',
+ 'pixiv file page api parser'
+ ] )
+
+ if do_twitter_stuff:
+
+ domain_manager.DeleteGUGs( [
+ 'twitter collection lookup',
+ 'twitter likes lookup',
+ 'twitter list lookup'
+ ] )
+
+ url_classes = domain_manager.GetURLClasses()
+
+ deletee_url_class_names = [ url_class.GetName() for url_class in url_classes if url_class.GetName() == 'twitter list' or url_class.GetName().startswith( 'twitter syndication api' ) ]
+
+ domain_manager.DeleteURLClasses( deletee_url_class_names )
+
+ # we're going to leave the one spare non-overwritten parser in place
+
+ #
+
+ domain_manager.OverwriteDefaultGUGs( [
+ 'twitter profile lookup',
+ 'twitter profile lookup (with replies)'
+ ] )
+
+ domain_manager.OverwriteDefaultURLClasses( [
+ 'twitter tweet (i/web/status)',
+ 'twitter tweet',
+ 'twitter syndication api tweet-result',
+ 'twitter syndication api timeline-profile'
+ ] )
+
+ domain_manager.OverwriteDefaultParsers( [
+ 'twitter syndication api timeline-profile parser',
+ 'twitter syndication api tweet parser'
+ ] )
+
+
+ domain_manager.TryToLinkURLClassesAndParsers()
+
+ #
+
+ self.modules_serialisable.SetJSONDump( domain_manager )
+
+ except Exception as e:
+
+ HydrusData.PrintException( e )
+
+ message = 'Trying to update some downloader objects failed! Please let hydrus dev know!'
+
+ self.pub_initial_message( message )
+
+
+
self._controller.frame_splash_status.SetTitleText( 'updated db to v{}'.format( HydrusData.ToHumanInt( version + 1 ) ) )
self._Execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
diff --git a/hydrus/client/db/ClientDBFilesStorage.py b/hydrus/client/db/ClientDBFilesStorage.py
index b4384f7d..d96a51cd 100644
--- a/hydrus/client/db/ClientDBFilesStorage.py
+++ b/hydrus/client/db/ClientDBFilesStorage.py
@@ -373,7 +373,7 @@ class ClientDBFilesStorage( ClientDBModule.ClientDBModule ):
service_ids_to_nums_cleared = {}
- local_non_trash_service_ids = self.modules_services.GetServiceIds( ( HC.COMBINED_LOCAL_FILE, HC.LOCAL_FILE_DOMAIN ) )
+ local_non_trash_service_ids = self.modules_services.GetServiceIds( ( HC.COMBINED_LOCAL_FILE, HC.COMBINED_LOCAL_MEDIA, HC.LOCAL_FILE_DOMAIN ) )
if hash_ids is None:
diff --git a/hydrus/client/gui/ClientGUIGallerySeedLog.py b/hydrus/client/gui/ClientGUIGallerySeedLog.py
index 21f61874..61a9af0e 100644
--- a/hydrus/client/gui/ClientGUIGallerySeedLog.py
+++ b/hydrus/client/gui/ClientGUIGallerySeedLog.py
@@ -447,17 +447,14 @@ class EditGallerySeedLogPanel( ClientGUIScrolledPanels.EditPanel ):
def _TrySelectedAgain( self, can_generate_more_pages ):
- new_gallery_seeds = []
-
gallery_seeds = self._list_ctrl.GetData( only_selected = True )
for gallery_seed in gallery_seeds:
- new_gallery_seeds.append( gallery_seed.GenerateRestartedDuplicate( can_generate_more_pages ) )
+ restarted_gallery_seed = gallery_seed.GenerateRestartedDuplicate( can_generate_more_pages )
+
+ self._gallery_seed_log.AddGallerySeeds( ( restarted_gallery_seed, ), parent_gallery_seed = gallery_seed )
-
- self._gallery_seed_log.AddGallerySeeds( new_gallery_seeds )
- self._gallery_seed_log.NotifyGallerySeedsUpdated( new_gallery_seeds )
def _UpdateListCtrl( self, gallery_seeds ):
diff --git a/hydrus/client/gui/ClientGUIScrolledPanelsManagement.py b/hydrus/client/gui/ClientGUIScrolledPanelsManagement.py
index cf46003e..f895704a 100644
--- a/hydrus/client/gui/ClientGUIScrolledPanelsManagement.py
+++ b/hydrus/client/gui/ClientGUIScrolledPanelsManagement.py
@@ -942,6 +942,10 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._delete_to_recycle_bin = QW.QCheckBox( self )
+ self._ms_to_wait_between_physical_file_deletes = ClientGUICommon.BetterSpinBox( self, min=20, max = 5000 )
+ tt = 'Deleting a file from a hard disk can be resource expensive, so when files leave the trash, the actual physical file delete happens later, in the background. The operation is spread out so as not to give you lag spikes.'
+ self._ms_to_wait_between_physical_file_deletes.setToolTip( tt )
+
self._confirm_trash = QW.QCheckBox( self )
self._confirm_archive = QW.QCheckBox( self )
@@ -987,6 +991,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._delete_to_recycle_bin.setChecked( HC.options[ 'delete_to_recycle_bin' ] )
+ self._ms_to_wait_between_physical_file_deletes.setValue( self._new_options.GetInteger( 'ms_to_wait_between_physical_file_deletes' ) )
+
self._confirm_trash.setChecked( HC.options[ 'confirm_trash' ] )
self._confirm_archive.setChecked( HC.options[ 'confirm_archive' ] )
@@ -1028,7 +1034,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
rows.append( ( 'Confirm sending more than one file to archive or inbox: ', self._confirm_archive ) )
rows.append( ( 'Confirm when copying files across local file services: ', self._confirm_multiple_local_file_services_copy ) )
rows.append( ( 'Confirm when moving files across local file services: ', self._confirm_multiple_local_file_services_move ) )
- rows.append( ( 'When deleting files or folders, send them to the OS\'s recycle bin: ', self._delete_to_recycle_bin ) )
+ rows.append( ( 'When physically deleting files or folders, send them to the OS\'s recycle bin: ', self._delete_to_recycle_bin ) )
+ rows.append( ( 'When maintenance physically deletes files, wait this many ms between each delete: ', self._ms_to_wait_between_physical_file_deletes ) )
rows.append( ( 'Remove files from view when they are filtered: ', self._remove_filtered_files ) )
rows.append( ( 'Remove files from view when they are sent to the trash: ', self._remove_trashed_files ) )
rows.append( ( 'Number of hours a file can be in the trash before being deleted: ', self._trash_max_age ) )
@@ -1120,6 +1127,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
HC.options[ 'trash_max_age' ] = self._trash_max_age.GetValue()
HC.options[ 'trash_max_size' ] = self._trash_max_size.GetValue()
+ self._new_options.SetInteger( 'ms_to_wait_between_physical_file_deletes', self._ms_to_wait_between_physical_file_deletes.value() )
+
self._new_options.SetBoolean( 'confirm_multiple_local_file_services_copy', self._confirm_multiple_local_file_services_copy.isChecked() )
self._new_options.SetBoolean( 'confirm_multiple_local_file_services_move', self._confirm_multiple_local_file_services_move.isChecked() )
@@ -3861,14 +3870,14 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
rows = []
- rows.append( ( 'Show related tags on single-file manage tags windows: ', self._show_related_tags ) )
+ rows.append( ( 'Show related tags: ', self._show_related_tags ) )
rows.append( ( 'Initial search duration (ms): ', self._related_tags_search_1_duration_ms ) )
rows.append( ( 'Medium search duration (ms): ', self._related_tags_search_2_duration_ms ) )
rows.append( ( 'Thorough search duration (ms): ', self._related_tags_search_3_duration_ms ) )
gridbox = ClientGUICommon.WrapInGrid( suggested_tags_related_panel, rows )
- desc = 'This will search the database for statistically related tags based on what your focused file already has.'
+ desc = 'This will search the database for tags statistically related to what your files already have.'
QP.AddToLayout( panel_vbox, ClientGUICommon.BetterStaticText(suggested_tags_related_panel,desc), CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( panel_vbox, gridbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
diff --git a/hydrus/client/gui/ClientGUIScrolledPanelsReview.py b/hydrus/client/gui/ClientGUIScrolledPanelsReview.py
index 0b93497f..0f6a4630 100644
--- a/hydrus/client/gui/ClientGUIScrolledPanelsReview.py
+++ b/hydrus/client/gui/ClientGUIScrolledPanelsReview.py
@@ -2450,15 +2450,6 @@ class ReviewFileHistory( ClientGUIScrolledPanels.ReviewPanel ):
vbox = QP.VBoxLayout()
- label = 'Please note that delete and inbox time tracking are new so you may not have full data for them.'
-
- st = ClientGUICommon.BetterStaticText( self, label = label )
-
- st.setWordWrap( True )
- st.setAlignment( QC.Qt.AlignCenter )
-
- QP.AddToLayout( vbox, st, CC.FLAGS_EXPAND_PERPENDICULAR )
-
flip_deleted = QW.QCheckBox( 'show deleted', self )
flip_deleted.setChecked( True )
@@ -3424,6 +3415,8 @@ class ReviewLocalFileImports( ClientGUIScrolledPanels.ReviewPanel ):
num_unimportable_mime_files = 0
num_occupied_files = 0
+ num_sidecars = 0
+
while not HG.started_shutdown:
if not self or not QP.isValid( self ):
@@ -3462,7 +3455,7 @@ class ReviewLocalFileImports( ClientGUIScrolledPanels.ReviewPanel ):
message = HydrusData.ConvertValueRangeToPrettyString( num_good_files, total_paths ) + ' files parsed successfully'
- if num_empty_files + num_missing_files + num_unimportable_mime_files + num_occupied_files > 0:
+ if num_empty_files + num_missing_files + num_unimportable_mime_files + num_occupied_files + num_sidecars > 0:
if num_good_files == 0:
@@ -3475,6 +3468,11 @@ class ReviewLocalFileImports( ClientGUIScrolledPanels.ReviewPanel ):
bad_comments = []
+ if num_sidecars > 0:
+
+ bad_comments.append( '{} looked like txt/json/xml sidecars'.format( HydrusData.ToHumanInt( num_sidecars ) ) )
+
+
if num_empty_files > 0:
bad_comments.append( HydrusData.ToHumanInt( num_empty_files ) + ' were empty' )
@@ -3544,9 +3542,10 @@ class ReviewLocalFileImports( ClientGUIScrolledPanels.ReviewPanel ):
try:
raw_paths = unparsed_paths_queue.get( block = False )
+
+ ( paths, num_sidecars_this_loop ) = ClientFiles.GetAllFilePaths( raw_paths, do_human_sort = do_human_sort ) # convert any dirs to subpaths
- paths = ClientFiles.GetAllFilePaths( raw_paths, do_human_sort = do_human_sort ) # convert any dirs to subpaths
-
+ num_sidecars += num_sidecars_this_loop
unparsed_paths.extend( paths )
except queue.Empty:
@@ -3562,9 +3561,10 @@ class ReviewLocalFileImports( ClientGUIScrolledPanels.ReviewPanel ):
try:
raw_paths = unparsed_paths_queue.get( timeout = 5 )
+
+ ( paths, num_sidecars_this_loop ) = ClientFiles.GetAllFilePaths( raw_paths, do_human_sort = do_human_sort ) # convert any dirs to subpaths
- paths = ClientFiles.GetAllFilePaths( raw_paths, do_human_sort = do_human_sort ) # convert any dirs to subpaths
-
+ num_sidecars += num_sidecars_this_loop
unparsed_paths.extend( paths )
except queue.Empty:
diff --git a/hydrus/client/gui/ClientGUISubscriptions.py b/hydrus/client/gui/ClientGUISubscriptions.py
index bf32f538..c63aab8e 100644
--- a/hydrus/client/gui/ClientGUISubscriptions.py
+++ b/hydrus/client/gui/ClientGUISubscriptions.py
@@ -189,7 +189,7 @@ class EditSubscriptionPanel( ClientGUIScrolledPanels.EditPanel ):
if HG.client_controller.new_options.GetBoolean( 'advanced_mode' ):
- limits_max = 10000
+ limits_max = 50000
else:
diff --git a/hydrus/client/gui/ClientGUITagSuggestions.py b/hydrus/client/gui/ClientGUITagSuggestions.py
index ed930361..3d61e85f 100644
--- a/hydrus/client/gui/ClientGUITagSuggestions.py
+++ b/hydrus/client/gui/ClientGUITagSuggestions.py
@@ -1,3 +1,4 @@
+import os
import typing
from qtpy import QtCore as QC
@@ -35,14 +36,18 @@ def FilterSuggestedPredicatesForMedia( predicates: typing.Sequence[ ClientSearch
def FilterSuggestedTagsForMedia( tags: typing.Sequence[ str ], medias: typing.Collection[ ClientMedia.Media ], service_key: bytes ) -> typing.List[ str ]:
+ # TODO: figure out a nice way to filter out siblings here
+ # maybe have to wait for when tags always know their siblings
+ # then we could also filter out worse/better siblings of the same count
+
+ num_media = len( medias )
+
tags_filtered_set = set( tags )
( current_tags_to_count, deleted_tags_to_count, pending_tags_to_count, petitioned_tags_to_count ) = ClientMedia.GetMediasTagCount( medias, service_key, ClientTags.TAG_DISPLAY_STORAGE )
current_tags_to_count.update( pending_tags_to_count )
- num_media = len( medias )
-
for ( tag, count ) in current_tags_to_count.items():
if count == num_media:
@@ -331,17 +336,46 @@ class RelatedTagsPanel( QW.QWidget ):
self._have_fetched = False
+ self._selected_tags = set()
+
self._new_options = HG.client_controller.new_options
vbox = QP.VBoxLayout()
+ tt = 'If you select some tags, this will search using only those as reference!'
+
self._button_2 = QW.QPushButton( 'medium', self )
self._button_2.clicked.connect( self.RefreshMedium )
self._button_2.setMinimumWidth( 30 )
+ self._button_2.setToolTip( tt )
self._button_3 = QW.QPushButton( 'thorough', self )
self._button_3.clicked.connect( self.RefreshThorough )
self._button_3.setMinimumWidth( 30 )
+ self._button_3.setToolTip( tt )
+
+ self._button_new = QW.QPushButton( 'new 1', self )
+ self._button_new.clicked.connect( self.RefreshNew )
+ self._button_new.setMinimumWidth( 30 )
+ tt = 'Please test this! This uses the new statistical method and searches your local files\' tags. Should be pretty fast, but its search domain is limited.' + os.linesep * 2 + 'Hydev thinks this mode sucks for the PTR, so let him know if it is actually works ok there.'
+ self._button_new.setToolTip( tt )
+
+ self._button_new_2 = QW.QPushButton( 'new 2', self )
+ self._button_new_2.clicked.connect( self.RefreshNew2 )
+ self._button_new_2.setMinimumWidth( 30 )
+ tt = 'Please test this! This uses the new statistical method and searches all the service\'s tags. May search slow and will not get results from large-count tags.' + os.linesep * 2 + 'Hydev wants to use this in the end, so let him know if it is too laggy.'
+ self._button_new_2.setToolTip( tt )
+
+ if len( self._media ) > 1:
+
+ self._button_2.setVisible( False )
+ self._button_3.setVisible( False )
+
+
+ if HG.client_controller.services_manager.GetServiceType( self._service_key ) == HC.LOCAL_TAG:
+
+ self._button_new_2.setVisible( False )
+
self._related_tags = ListBoxTagsSuggestionsRelated( self, service_key, activate_callable )
@@ -349,6 +383,8 @@ class RelatedTagsPanel( QW.QWidget ):
QP.AddToLayout( button_hbox, self._button_2, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
QP.AddToLayout( button_hbox, self._button_3, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
+ QP.AddToLayout( button_hbox, self._button_new, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
+ QP.AddToLayout( button_hbox, self._button_new_2, CC.FLAGS_EXPAND_SIZER_BOTH_WAYS )
QP.AddToLayout( vbox, button_hbox, CC.FLAGS_EXPAND_PERPENDICULAR )
QP.AddToLayout( vbox, self._related_tags, CC.FLAGS_EXPAND_BOTH_WAYS )
@@ -385,17 +421,77 @@ class RelatedTagsPanel( QW.QWidget ):
self._related_tags.SetPredicates( [] )
+ if len( self._media ) > 1:
+
+ return
+
+
( m, ) = self._media
hash = m.GetHash()
- search_tags = ClientMedia.GetMediasTags( self._media, self._service_key, ClientTags.TAG_DISPLAY_STORAGE, ( HC.CONTENT_STATUS_CURRENT, HC.CONTENT_STATUS_PENDING ) )
+ # TODO: If user has some tags selected, use them instead
+
+ if len( self._selected_tags ) == 0:
+
+ search_tags = ClientMedia.GetMediasTags( self._media, self._service_key, ClientTags.TAG_DISPLAY_STORAGE, ( HC.CONTENT_STATUS_CURRENT, HC.CONTENT_STATUS_PENDING ) )
+
+ else:
+
+ search_tags = self._selected_tags
+
max_results = 100
HG.client_controller.CallToThread( do_it, self._service_key, hash, search_tags, max_results, max_time_to_take )
+ def _FetchRelatedTagsNew( self, file_service_key = None ):
+
+ def do_it( file_service_key, tag_service_key, search_tags ):
+
+ def qt_code( predicates ):
+
+ if not self or not QP.isValid( self ):
+
+ return
+
+
+ self._last_fetched_predicates = predicates
+
+ self._UpdateTagDisplay()
+
+ self._have_fetched = True
+
+
+ predicates = HG.client_controller.Read( 'related_tags_new', file_service_key, tag_service_key, search_tags )
+
+ predicates = ClientSearch.SortPredicates( predicates )
+
+ QP.CallAfter( qt_code, predicates )
+
+
+ self._related_tags.SetPredicates( [] )
+
+ if len( self._selected_tags ) == 0:
+
+ search_tags = ClientMedia.GetMediasTags( self._media, self._service_key, ClientTags.TAG_DISPLAY_STORAGE, ( HC.CONTENT_STATUS_CURRENT, HC.CONTENT_STATUS_PENDING ) )
+
+ else:
+
+ search_tags = self._selected_tags
+
+
+ if file_service_key is None:
+
+ file_service_key = CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY
+
+
+ tag_service_key = self._service_key
+
+ HG.client_controller.CallToThread( do_it, file_service_key, tag_service_key, search_tags )
+
+
def _QuickSuggestedRelatedTags( self ):
max_time_to_take = self._new_options.GetInteger( 'related_tags_search_1_duration_ms' ) / 1000.0
@@ -424,6 +520,16 @@ class RelatedTagsPanel( QW.QWidget ):
self._FetchRelatedTags( max_time_to_take )
+ def RefreshNew( self ):
+
+ self._FetchRelatedTagsNew()
+
+
+ def RefreshNew2( self ):
+
+ self._FetchRelatedTagsNew( file_service_key = CC.COMBINED_FILE_SERVICE_KEY )
+
+
def MediaUpdated( self ):
self._UpdateTagDisplay()
@@ -433,7 +539,19 @@ class RelatedTagsPanel( QW.QWidget ):
self._media = media
- self._QuickSuggestedRelatedTags()
+ if len( self._media ) == 1:
+
+ self._QuickSuggestedRelatedTags()
+
+ else:
+
+ self._related_tags.SetPredicates( [] )
+
+
+
+ def SetSelectedTags( self, tags ):
+
+ self._selected_tags = tags
def TakeFocusForUser( self ):
@@ -682,7 +800,7 @@ class SuggestedTagsPanel( QW.QWidget ):
self._related_tags = None
- if self._new_options.GetBoolean( 'show_related_tags' ) and len( media ) == 1:
+ if self._new_options.GetBoolean( 'show_related_tags' ):
self._related_tags = RelatedTagsPanel( panel_parent, service_key, media, activate_callable )
@@ -796,6 +914,14 @@ class SuggestedTagsPanel( QW.QWidget ):
+ def SetSelectedTags( self, tags ):
+
+ if self._related_tags is not None:
+
+ self._related_tags.SetSelectedTags( tags )
+
+
+
def TakeFocusForUser( self, command ):
if command == CAC.SIMPLE_SHOW_AND_FOCUS_MANAGE_TAGS_FAVOURITE_TAGS:
diff --git a/hydrus/client/gui/ClientGUITags.py b/hydrus/client/gui/ClientGUITags.py
index 4a383aea..5747d981 100644
--- a/hydrus/client/gui/ClientGUITags.py
+++ b/hydrus/client/gui/ClientGUITags.py
@@ -2298,6 +2298,8 @@ class ManageTagsPanel( ClientGUIScrolledPanels.ManagePanel, CAC.ApplicationComma
self._suggested_tags.mouseActivationOccurred.connect( self.SetTagBoxFocus )
+ self._tags_box.tagsSelected.connect( self._suggested_tags.SetSelectedTags )
+
def _EnterTags( self, tags, only_add = False, only_remove = False, forced_reason = None ):
@@ -4473,18 +4475,33 @@ class ManageTagSiblings( ClientGUIScrolledPanels.ManagePanel ):
current_dict = dict( current_pairs )
- current_olds = set( current_dict.keys() )
-
for ( potential_old, potential_new ) in pairs:
if potential_new in current_dict:
loop_new = potential_new
+ seen_tags = set()
+
while loop_new in current_dict:
+ seen_tags.add( loop_new )
+
next_new = current_dict[ loop_new ]
+ if next_new in seen_tags:
+
+ QW.QMessageBox.warning( self, 'Loop Problem!', 'While trying to test the new pair(s) for potential loops, I think I ran across an existing loop! Please review everything and see if you can break any loops yourself.' )
+
+ message = 'The pair you mean to add seems to connect to a sibling loop already in your database! Please undo this loop manually. The tags involved in the loop are:'
+ message += os.linesep * 2
+ message += ', '.join( seen_tags )
+
+ QW.QMessageBox.critical( self, 'Error', message )
+
+ break
+
+
if next_new == potential_old:
pairs_to_auto_petition = [ ( loop_new, next_new ) ]
diff --git a/hydrus/client/gui/canvas/ClientGUICanvas.py b/hydrus/client/gui/canvas/ClientGUICanvas.py
index 7702c171..892eca3c 100644
--- a/hydrus/client/gui/canvas/ClientGUICanvas.py
+++ b/hydrus/client/gui/canvas/ClientGUICanvas.py
@@ -2098,7 +2098,7 @@ class CanvasWithHovers( CanvasWithDetails ):
# due to the mouse setPos below, the event pos can get funky I think due to out of order coordinate setting events, so we'll poll current value directly
event_pos = self.mapFromGlobal( QG.QCursor.pos() )
- mouse_currently_shown = self.cursor() == QG.QCursor( QC.Qt.ArrowCursor )
+ mouse_currently_shown = self.cursor().shape() == QC.Qt.ArrowCursor
show_mouse = mouse_currently_shown
is_dragging = event.buttons() & QC.Qt.LeftButton and self._last_drag_pos is not None
diff --git a/hydrus/client/gui/exporting/ClientGUIExport.py b/hydrus/client/gui/exporting/ClientGUIExport.py
index c400118a..40734319 100644
--- a/hydrus/client/gui/exporting/ClientGUIExport.py
+++ b/hydrus/client/gui/exporting/ClientGUIExport.py
@@ -83,6 +83,27 @@ class EditExportFoldersPanel( ClientGUIScrolledPanels.EditPanel ):
metadata_routers = new_options.GetDefaultExportFilesMetadataRouters()
+ if len( metadata_routers ) > 0:
+
+ message = 'You have some default metadata sidecar settings, most likely from a previous file export. They look like this:'
+ message += os.linesep * 2
+ message += os.linesep.join( [ router.ToString( pretty = True ) for router in metadata_routers ] )
+ message += os.linesep * 2
+ message += 'Do you want these in the new export folder?'
+
+ ( result, cancelled ) = ClientGUIDialogsQuick.GetYesNo( self, message, no_label = 'no, I want an empty sidecar list', check_for_cancelled = True )
+
+ if cancelled:
+
+ return
+
+
+ if result != QW.QDialog.DialogCode.Accepted:
+
+ metadata_routers = []
+
+
+
period = 15 * 60
export_folder = ClientExportingFiles.ExportFolder(
diff --git a/hydrus/client/gui/lists/ClientGUIListBoxes.py b/hydrus/client/gui/lists/ClientGUIListBoxes.py
index 747b7e64..009672f0 100644
--- a/hydrus/client/gui/lists/ClientGUIListBoxes.py
+++ b/hydrus/client/gui/lists/ClientGUIListBoxes.py
@@ -1489,6 +1489,8 @@ class ListBox( QW.QScrollArea ):
+ self._SelectionChanged()
+
self.widget().update()
@@ -1781,6 +1783,11 @@ class ListBox( QW.QScrollArea ):
self._selected_terms = set( self._terms_to_logical_indices.keys() )
+ def _SelectionChanged( self ):
+
+ pass
+
+
def _SetVirtualSize( self ):
self.setWidgetResizable( True )
@@ -2169,6 +2176,7 @@ COPY_ALL_SUBTAGS_WITH_COUNTS = 7
class ListBoxTags( ListBox ):
+ tagsSelected = QC.Signal( set )
can_spawn_new_windows = True
def __init__( self, parent, *args, tag_display_type: int = ClientTags.TAG_DISPLAY_STORAGE, **kwargs ):
@@ -2437,6 +2445,13 @@ class ListBoxTags( ListBox ):
pass
+ def _SelectionChanged( self ):
+
+ tags = set( self._GetTagsFromTerms( self._selected_terms ) )
+
+ self.tagsSelected.emit( tags )
+
+
def _UpdateBackgroundColour( self ):
new_options = HG.client_controller.new_options
diff --git a/hydrus/client/gui/pages/ClientGUIManagement.py b/hydrus/client/gui/pages/ClientGUIManagement.py
index 861e3739..9f16a001 100644
--- a/hydrus/client/gui/pages/ClientGUIManagement.py
+++ b/hydrus/client/gui/pages/ClientGUIManagement.py
@@ -996,6 +996,8 @@ class ManagementPanel( QW.QScrollArea ):
self._page = page
self._page_key = self._management_controller.GetVariable( 'page_key' )
+ self._page_state = CC.PAGE_STATE_NORMAL
+
self._current_selection_tags_list = None
self._media_sort = ClientGUIResultsSortCollect.MediaSortControl( self, management_controller = self._management_controller )
@@ -1097,6 +1099,11 @@ class ManagementPanel( QW.QScrollArea ):
return media_panel
+ def GetPageState( self ) -> int:
+
+ return self._page_state
+
+
def PageHidden( self ):
pass
@@ -1620,16 +1627,20 @@ class ManagementPanelDuplicateFilter( ManagementPanel ):
self._page.SwapMediaPanel( panel )
+ self._page_state = CC.PAGE_STATE_NORMAL
+
def _ShowRandomPotentialDupes( self ):
( file_search_context_1, file_search_context_2, dupe_search_type, pixel_dupes_preference, max_hamming_distance ) = self._GetDuplicateFileSearchData()
+ self._page_state = CC.PAGE_STATE_SEARCHING
+
hashes = self._controller.Read( 'random_potential_duplicate_hashes', file_search_context_1, file_search_context_2, dupe_search_type, pixel_dupes_preference, max_hamming_distance )
if len( hashes ) == 0:
- HydrusData.ShowText( 'No files were found. Try refreshing the count, and if this keeps happening, please let hydrus_dev know.' )
+ HydrusData.ShowText( 'No random potential duplicates were found. Try refreshing the count, and if this keeps happening, please let hydrus_dev know.' )
self._ShowPotentialDupes( hashes )
@@ -5501,6 +5512,8 @@ class ManagementPanelQuery( ManagementPanel ):
self._page.SwapMediaPanel( panel )
+ self._page_state = CC.PAGE_STATE_SEARCHING_CANCELLED
+
self._UpdateCancelButton()
@@ -5576,6 +5589,8 @@ class ManagementPanelQuery( ManagementPanel ):
panel = ClientGUIResults.MediaPanelLoading( self._page, self._page_key, location_context )
+ self._page_state = CC.PAGE_STATE_SEARCHING
+
else:
panel = ClientGUIResults.MediaPanelThumbnails( self._page, self._page_key, location_context, [] )
@@ -5749,6 +5764,8 @@ class ManagementPanelQuery( ManagementPanel ):
self._page.SwapMediaPanel( panel )
+ self._page_state = CC.PAGE_STATE_NORMAL
+
def Start( self ):
diff --git a/hydrus/client/gui/pages/ClientGUIPages.py b/hydrus/client/gui/pages/ClientGUIPages.py
index c26ff0c3..ac861f7d 100644
--- a/hydrus/client/gui/pages/ClientGUIPages.py
+++ b/hydrus/client/gui/pages/ClientGUIPages.py
@@ -664,6 +664,7 @@ class Page( QW.QWidget ):
d[ 'name' ] = self._management_controller.GetPageName()
d[ 'page_key' ] = self._page_key.hex()
+ d[ 'page_state' ] = self.GetPageState()
d[ 'page_type' ] = self._management_controller.GetType()
management_info = self._management_controller.GetAPIInfoDict( simple )
@@ -770,6 +771,18 @@ class Page( QW.QWidget ):
return { self._page_key }
+ def GetPageState( self ) -> int:
+
+ if self._initialised:
+
+ return self._management_panel.GetPageState()
+
+ else:
+
+ return CC.PAGE_STATE_INITIALISING
+
+
+
def GetParentNotebook( self ):
return self._parent_notebook
@@ -819,6 +832,7 @@ class Page( QW.QWidget ):
root[ 'name' ] = self.GetName()
root[ 'page_key' ] = self._page_key.hex()
+ root[ 'page_state' ] = self.GetPageState()
root[ 'page_type' ] = self._management_controller.GetType()
root[ 'selected' ] = is_selected
@@ -2286,7 +2300,12 @@ class PagesNotebook( QP.TabWidgetWithDnD ):
def GetAPIInfoDict( self, simple ):
- return {}
+ return {
+ 'name' : self.GetName(),
+ 'page_key' : self._page_key.hex(),
+ 'page_state' : self.GetPageState(),
+ 'page_type' : ClientGUIManagement.MANAGEMENT_TYPE_PAGE_OF_PAGES
+ }
def GetCurrentGUISession( self, name: str, only_changed_page_data: bool, about_to_save: bool ):
@@ -2540,6 +2559,11 @@ class PagesNotebook( QP.TabWidgetWithDnD ):
return self._GetPages()
+ def GetPageState( self ) -> int:
+
+ return CC.PAGE_STATE_NORMAL
+
+
def GetPrettyStatusForStatusBar( self ):
( num_files, ( num_value, num_range ) ) = self.GetNumFileSummary()
@@ -2596,6 +2620,7 @@ class PagesNotebook( QP.TabWidgetWithDnD ):
root[ 'name' ] = self.GetName()
root[ 'page_key' ] = self._page_key.hex()
+ root[ 'page_state' ] = self.GetPageState()
root[ 'page_type' ] = ClientGUIManagement.MANAGEMENT_TYPE_PAGE_OF_PAGES
root[ 'selected' ] = is_selected
root[ 'pages' ] = my_pages_list
diff --git a/hydrus/client/gui/search/ClientGUIACDropdown.py b/hydrus/client/gui/search/ClientGUIACDropdown.py
index 7aea7d0e..36ce4cea 100644
--- a/hydrus/client/gui/search/ClientGUIACDropdown.py
+++ b/hydrus/client/gui/search/ClientGUIACDropdown.py
@@ -43,7 +43,9 @@ def InsertOtherPredicatesForRead( predicates: list, parsed_autocomplete_text: Cl
if include_unusual_predicate_types:
- non_tag_predicates = list( parsed_autocomplete_text.GetNonTagFileSearchPredicates() )
+ allow_auto_wildcard_conversion = True
+
+ non_tag_predicates = list( parsed_autocomplete_text.GetNonTagFileSearchPredicates( allow_auto_wildcard_conversion ) )
non_tag_predicates.reverse()
@@ -58,11 +60,11 @@ def InsertOtherPredicatesForRead( predicates: list, parsed_autocomplete_text: Cl
PutAtTopOfMatches( predicates, under_construction_or_predicate )
-def InsertTagPredicates( predicates: list, tag_service_key: bytes, parsed_autocomplete_text: ClientSearch.ParsedAutocompleteText, insert_if_does_not_exist: bool = True ):
+def InsertTagPredicates( predicates: list, tag_service_key: bytes, parsed_autocomplete_text: ClientSearch.ParsedAutocompleteText, allow_auto_wildcard_conversion: bool, insert_if_does_not_exist: bool = True ):
- if parsed_autocomplete_text.IsTagSearch():
+ if parsed_autocomplete_text.IsTagSearch( allow_auto_wildcard_conversion):
- tag_predicate = parsed_autocomplete_text.GetImmediateFileSearchPredicate()
+ tag_predicate = parsed_autocomplete_text.GetImmediateFileSearchPredicate( allow_auto_wildcard_conversion )
actual_tag = tag_predicate.GetValue()
@@ -185,7 +187,9 @@ def ReadFetch(
if fetch_from_db:
- is_explicit_wildcard = parsed_autocomplete_text.IsExplicitWildcard()
+ allow_auto_wildcard_conversion = True
+
+ is_explicit_wildcard = parsed_autocomplete_text.IsExplicitWildcard( allow_auto_wildcard_conversion )
small_exact_match_search = ShouldDoExactSearch( parsed_autocomplete_text )
@@ -340,7 +344,9 @@ def ReadFetch(
- InsertTagPredicates( matches, tag_service_key, parsed_autocomplete_text, insert_if_does_not_exist = False )
+ allow_auto_wildcard_conversion = True
+
+ InsertTagPredicates( matches, tag_service_key, parsed_autocomplete_text, allow_auto_wildcard_conversion, insert_if_does_not_exist = False )
InsertOtherPredicatesForRead( matches, parsed_autocomplete_text, include_unusual_predicate_types, under_construction_or_predicate )
@@ -376,7 +382,9 @@ def PutAtTopOfMatches( matches: list, predicate: ClientSearch.Predicate, insert_
def ShouldDoExactSearch( parsed_autocomplete_text: ClientSearch.ParsedAutocompleteText ):
- if parsed_autocomplete_text.IsExplicitWildcard():
+ allow_auto_wildcard_conversion = True
+
+ if parsed_autocomplete_text.IsExplicitWildcard( allow_auto_wildcard_conversion ):
return False
@@ -418,9 +426,12 @@ def WriteFetch( win, job_key, results_callable, parsed_autocomplete_text: Client
else:
- is_explicit_wildcard = parsed_autocomplete_text.IsExplicitWildcard()
+ allow_auto_wildcard_conversion = False
- strict_search_text = parsed_autocomplete_text.GetSearchText( False )
+ # TODO: This allow_auto_wildcard_conversion hack to handle allow_unnamespaced_search_gives_any_namespace_wildcards is hell. I should write IsImplicitWildcard or something!
+ is_explicit_wildcard = parsed_autocomplete_text.IsExplicitWildcard( allow_auto_wildcard_conversion )
+
+ strict_search_text = parsed_autocomplete_text.GetSearchText( False, allow_auto_wildcard_conversion = allow_auto_wildcard_conversion )
autocomplete_search_text = parsed_autocomplete_text.GetSearchText( True )
small_exact_match_search = ShouldDoExactSearch( parsed_autocomplete_text )
@@ -490,7 +501,9 @@ def WriteFetch( win, job_key, results_callable, parsed_autocomplete_text: Client
matches = ClientSearch.SortPredicates( matches )
- InsertTagPredicates( matches, display_tag_service_key, parsed_autocomplete_text )
+ allow_auto_wildcard_conversion = False
+
+ InsertTagPredicates( matches, display_tag_service_key, parsed_autocomplete_text, allow_auto_wildcard_conversion )
HG.client_controller.CallAfterQtSafe( win, 'write a/c fetch', results_callable, job_key, parsed_autocomplete_text, results_cache, matches )
@@ -1909,7 +1922,9 @@ class AutoCompleteDropdownTagsRead( AutoCompleteDropdownTags ):
if parsed_autocomplete_text.IsAcceptableForFileSearches():
- return parsed_autocomplete_text.GetImmediateFileSearchPredicate()
+ allow_auto_wildcard_conversion = True
+
+ return parsed_autocomplete_text.GetImmediateFileSearchPredicate( allow_auto_wildcard_conversion )
else:
@@ -2656,9 +2671,11 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
parsed_autocomplete_text = self._GetParsedAutocompleteText()
- if parsed_autocomplete_text.IsTagSearch():
+ allow_auto_wildcard_conversion = False
+
+ if parsed_autocomplete_text.IsTagSearch( allow_auto_wildcard_conversion ):
- return parsed_autocomplete_text.GetImmediateFileSearchPredicate()
+ return parsed_autocomplete_text.GetImmediateFileSearchPredicate( allow_auto_wildcard_conversion )
else:
@@ -2765,7 +2782,9 @@ class AutoCompleteDropdownTagsWrite( AutoCompleteDropdownTags ):
stub_predicates = []
- InsertTagPredicates( stub_predicates, self._display_tag_service_key, parsed_autocomplete_text )
+ allow_auto_wildcard_conversion = False
+
+ InsertTagPredicates( stub_predicates, self._display_tag_service_key, parsed_autocomplete_text, allow_auto_wildcard_conversion )
AppendLoadingPredicate( stub_predicates )
diff --git a/hydrus/client/gui/services/ClientGUIClientsideServices.py b/hydrus/client/gui/services/ClientGUIClientsideServices.py
index a97d7b68..03a3ad8b 100644
--- a/hydrus/client/gui/services/ClientGUIClientsideServices.py
+++ b/hydrus/client/gui/services/ClientGUIClientsideServices.py
@@ -1602,7 +1602,6 @@ class ReviewServicePanel( QW.QWidget ):
if not HG.client_controller.new_options.GetBoolean( 'advanced_mode' ):
self._id_button.hide()
- self._service_key_button.hide()
vbox = QP.VBoxLayout()
diff --git a/hydrus/client/importing/ClientImportFileSeeds.py b/hydrus/client/importing/ClientImportFileSeeds.py
index 0bf4d4ea..5818455a 100644
--- a/hydrus/client/importing/ClientImportFileSeeds.py
+++ b/hydrus/client/importing/ClientImportFileSeeds.py
@@ -1,4 +1,3 @@
-import bisect
import collections
import itertools
import os
@@ -1921,8 +1920,61 @@ class FileSeedCacheStatus( HydrusSerialisable.SerialisableBase ):
self._latest_added_time = latest_added_time
+
HydrusSerialisable.SERIALISABLE_TYPES_TO_OBJECT_TYPES[ HydrusSerialisable.SERIALISABLE_TYPE_FILE_SEED_CACHE_STATUS ] = FileSeedCacheStatus
+def WalkToNextFileSeed( status, starting_file_seed, file_seeds, file_seeds_to_indices, statuses_to_file_seeds ):
+
+ # the file seed given is the starting point and can be the answer
+ # but if it is wrong, or no longer tracked, then let's walk through them until we get one, or None
+ # along the way, we'll note what's in the wrong place
+
+ wrong_file_seeds = set()
+
+ if starting_file_seed in file_seeds_to_indices:
+
+ index = file_seeds_to_indices[ starting_file_seed ]
+
+ else:
+
+ index = 0
+
+
+ file_seed = starting_file_seed
+
+ while True:
+
+ # no need to walk further, we are good
+
+ if file_seed.status == status:
+
+ result = file_seed
+
+ break
+
+
+ # this file seed has the wrong status, move on
+
+ if file_seed in statuses_to_file_seeds[ status ]:
+
+ wrong_file_seeds.add( file_seed )
+
+
+ index += 1
+
+ if index >= len( file_seeds ):
+
+ result = None
+
+ break
+
+
+ file_seed = file_seeds[ index ]
+
+
+ return ( result, wrong_file_seeds )
+
+
class FileSeedCache( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_FILE_SEED_CACHE
@@ -1939,14 +1991,21 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
self._file_seeds_to_indices = {}
- self._statuses_to_indexed_file_seeds = collections.defaultdict( list )
+ # if there's a file seed here, it is the earliest
+ # if there's none in here, we know the count is 0
+ # if status is absent, we don't know
+ self._observed_statuses_to_next_file_seeds = {}
+
+ self._file_seeds_to_observed_statuses = {}
+ self._statuses_to_file_seeds = collections.defaultdict( set )
self._file_seed_cache_key = HydrusData.GenerateKey()
self._status_cache = FileSeedCacheStatus()
self._status_dirty = True
- self._statuses_to_indexed_file_seeds_dirty = True
+ self._statuses_to_file_seeds_dirty = True
+ self._file_seeds_to_indices_dirty = True
self._lock = threading.Lock()
@@ -1956,56 +2015,88 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
return len( self._file_seeds )
- def _FileSeedIndicesJustChanged( self ):
+ def _FixStatusesToFileSeeds( self, file_seeds: typing.Collection[ FileSeed ] ):
- self._file_seeds_to_indices = { file_seed : index for ( index, file_seed ) in enumerate( self._file_seeds ) }
+ if self._statuses_to_file_seeds_dirty:
+
+ return
+
- self._SetStatusesToFileSeedsDirty()
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
+ statuses_to_file_seeds = self._GetStatusesToFileSeeds()
-
- def _FixFileSeedsStatusPosition( self, file_seeds ):
+ if len( file_seeds ) == 0:
+
+ return
+
- indices_and_file_seeds_affected = []
+ outstanding_others_to_fix = set()
for file_seed in file_seeds:
- if file_seed in self._file_seeds_to_indices:
-
- indices_and_file_seeds_affected.append( ( self._file_seeds_to_indices[ file_seed ], file_seed ) )
-
- else:
-
- self._SetStatusesToFileSeedsDirty()
-
- return
-
+ want_a_record = file_seed in file_seeds_to_indices
+ record_exists = file_seed in self._file_seeds_to_observed_statuses
-
- for row in indices_and_file_seeds_affected:
-
- correct_status = row[1].status
-
- if row in self._statuses_to_indexed_file_seeds[ correct_status ]:
+ if not want_a_record and not record_exists:
continue
- for ( status, indices_and_file_seeds ) in self._statuses_to_indexed_file_seeds.items():
+ correct_status = file_seed.status
+
+ if record_exists:
- if status == correct_status:
+ set_status = self._file_seeds_to_observed_statuses[ file_seed ]
+
+ if set_status != correct_status or not want_a_record:
- continue
+ if set_status in self._observed_statuses_to_next_file_seeds:
+
+ if self._observed_statuses_to_next_file_seeds[ set_status ] == file_seed:
+
+ # this 'next' is now wrong, so fast forward to the correct one, or None
+ ( result, wrong_file_seeds ) = WalkToNextFileSeed( set_status, file_seed, self._file_seeds, file_seeds_to_indices, statuses_to_file_seeds )
+
+ self._observed_statuses_to_next_file_seeds[ set_status ] = result
+
+ outstanding_others_to_fix.update( wrong_file_seeds )
+
+
+
+ statuses_to_file_seeds[ set_status ].discard( file_seed )
+
+ del self._file_seeds_to_observed_statuses[ file_seed ]
+
+ record_exists = False
- if row in indices_and_file_seeds:
+
+ if want_a_record:
+
+ if not record_exists:
- indices_and_file_seeds.remove( row )
+ statuses_to_file_seeds[ correct_status ].add( file_seed )
- bisect.insort( self._statuses_to_indexed_file_seeds[ correct_status ], row )
-
- break
+ self._file_seeds_to_observed_statuses[ file_seed ] = correct_status
+ if correct_status in self._observed_statuses_to_next_file_seeds:
+
+ current_next_file_seed = self._observed_statuses_to_next_file_seeds[ correct_status ]
+
+ if current_next_file_seed is None or file_seeds_to_indices[ file_seed ] < file_seeds_to_indices[ current_next_file_seed ]:
+
+ self._observed_statuses_to_next_file_seeds[ correct_status ] = file_seed
+
+
+
+
+
+ outstanding_others_to_fix.difference_update( file_seeds )
+
+ if len( outstanding_others_to_fix ) > 0:
+
+ self._FixStatusesToFileSeeds( outstanding_others_to_fix )
@@ -2029,15 +2120,25 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
else:
- if self._statuses_to_indexed_file_seeds_dirty:
-
- self._RegenerateStatusesToFileSeeds()
-
+ statuses_to_file_seeds = self._GetStatusesToFileSeeds()
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
- return [ file_seed for ( index, file_seed ) in self._statuses_to_indexed_file_seeds[ status ] ]
+ return sorted( statuses_to_file_seeds[ status ], key = lambda f_s: file_seeds_to_indices[ f_s ] )
+ def _GetFileSeedsToIndices( self ) -> typing.Dict[ FileSeed, int ]:
+
+ if self._file_seeds_to_indices_dirty:
+
+ self._file_seeds_to_indices = { file_seed : index for ( index, file_seed ) in enumerate( self._file_seeds ) }
+
+ self._file_seeds_to_indices_dirty = False
+
+
+ return self._file_seeds_to_indices
+
+
def _GetLatestAddedTime( self ):
if len( self._file_seeds ) == 0:
@@ -2069,36 +2170,41 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
def _GetNextFileSeed( self, status: int ) -> typing.Optional[ FileSeed ]:
+ statuses_to_file_seeds = self._GetStatusesToFileSeeds()
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
+
# the problem with this is if a file seed recently changed but 'notifyupdated' hasn't had a chance to go yet
# there could be a FS in a list other than the one we are looking at that has the status we want
- # _however_, it seems like I do not do any async calls to notifyupdated in the actual FSC, only from notifyupdated to GUI elements, so we _seem_ to be good
+ # _however_, it seems like I do not do any async calls to notifyupdated in the actual FSC, only from notifyupdated to GUI elements, so we _seem_ to be good to talk to this in this way
- if self._statuses_to_indexed_file_seeds_dirty:
+ if status not in self._observed_statuses_to_next_file_seeds:
- self._RegenerateStatusesToFileSeeds()
+ file_seeds = statuses_to_file_seeds[ status ]
-
- indexed_file_seeds = self._statuses_to_indexed_file_seeds[ status ]
-
- while len( indexed_file_seeds ) > 0:
-
- row = indexed_file_seeds[ 0 ]
-
- file_seed = row[1]
-
- if file_seed.status == status:
+ if len( file_seeds ) == 0:
- return file_seed
+ self._observed_statuses_to_next_file_seeds[ status ] = None
else:
- self._FixFileSeedsStatusPosition( ( file_seed, ) )
+ self._observed_statuses_to_next_file_seeds[ status ] = min( file_seeds, key = lambda f_s: file_seeds_to_indices[ f_s ] )
- indexed_file_seeds = self._statuses_to_indexed_file_seeds[ status ]
+
+ file_seed = self._observed_statuses_to_next_file_seeds[ status ]
+
+ if file_seed is None:
+
+ return None
- return None
+ ( result, wrong_file_seeds ) = WalkToNextFileSeed( status, file_seed, self._file_seeds, file_seeds_to_indices, statuses_to_file_seeds )
+
+ self._observed_statuses_to_next_file_seeds[ status ] = result
+
+ self._FixStatusesToFileSeeds( wrong_file_seeds )
+
+ return file_seed
def _GetSerialisableInfo( self ):
@@ -2125,14 +2231,11 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
statuses_to_counts = collections.Counter()
- if self._statuses_to_indexed_file_seeds_dirty:
-
- self._RegenerateStatusesToFileSeeds()
-
+ statuses_to_file_seeds = self._GetStatusesToFileSeeds()
- for ( status, indexed_file_seeds ) in self._statuses_to_indexed_file_seeds.items():
+ for ( status, file_seeds ) in statuses_to_file_seeds.items():
- count = len( indexed_file_seeds )
+ count = len( file_seeds )
if count > 0:
@@ -2143,11 +2246,51 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
return statuses_to_counts
+ def _GetStatusesToFileSeeds( self ) -> typing.Dict[ int, typing.Set[ FileSeed ] ]:
+
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
+
+ if self._statuses_to_file_seeds_dirty:
+
+ self._file_seeds_to_observed_statuses = {}
+ self._statuses_to_file_seeds = collections.defaultdict( set )
+ self._observed_statuses_to_next_file_seeds = {}
+
+ for ( file_seed, index ) in file_seeds_to_indices.items():
+
+ status = file_seed.status
+
+ self._statuses_to_file_seeds[ status ].add( file_seed )
+ self._file_seeds_to_observed_statuses[ file_seed ] = status
+
+ if status not in self._observed_statuses_to_next_file_seeds:
+
+ self._observed_statuses_to_next_file_seeds[ status ] = file_seed
+
+ else:
+
+ current_next = self._observed_statuses_to_next_file_seeds[ status ]
+
+ if current_next is not None and index < file_seeds_to_indices[ current_next ]:
+
+ self._observed_statuses_to_next_file_seeds[ status ] = file_seed
+
+
+
+
+ self._statuses_to_file_seeds_dirty = False
+
+
+ return self._statuses_to_file_seeds
+
+
def _HasFileSeed( self, file_seed: FileSeed ):
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
+
search_file_seeds = file_seed.GetSearchFileSeeds()
- has_file_seed = True in ( search_file_seed in self._file_seeds_to_indices for search_file_seed in search_file_seeds )
+ has_file_seed = True in ( search_file_seed in file_seeds_to_indices for search_file_seed in search_file_seeds )
return has_file_seed
@@ -2158,30 +2301,30 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
self._file_seeds = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_info )
- self._FileSeedIndicesJustChanged()
-
- def _RegenerateStatusesToFileSeeds( self ):
+ def _NotifyFileSeedsUpdated( self, file_seeds: typing.Collection[ FileSeed ] ):
- self._statuses_to_indexed_file_seeds = collections.defaultdict( list )
-
- for ( file_seed, index ) in self._file_seeds_to_indices.items():
+ if len( file_seeds ) == 0:
- self._statuses_to_indexed_file_seeds[ file_seed.status ].append( ( index, file_seed ) )
+ return
- for indexed_file_seeds in self._statuses_to_indexed_file_seeds.values():
-
- indexed_file_seeds.sort()
-
+ HG.client_controller.pub( 'file_seed_cache_file_seeds_updated', self._file_seed_cache_key, file_seeds )
- self._statuses_to_indexed_file_seeds_dirty = False
+
+ def _SetFileSeedsToIndicesDirty( self ):
+
+ self._file_seeds_to_indices_dirty = True
+
+ self._observed_statuses_to_next_file_seeds = {}
def _SetStatusesToFileSeedsDirty( self ):
- self._statuses_to_indexed_file_seeds_dirty = True
+ # this is never actually called, which is neat! I think we are 'perfect' on this thing maintaining itself after inital generation
+
+ self._statuses_to_file_seeds_dirty = True
def _SetStatusDirty( self ):
@@ -2406,6 +2549,8 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
with self._lock:
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
+
for file_seed in file_seeds:
if self._HasFileSeed( file_seed ):
@@ -2445,42 +2590,50 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
index = len( self._file_seeds ) - 1
- self._file_seeds_to_indices[ file_seed ] = index
-
- if not self._statuses_to_indexed_file_seeds_dirty:
-
- self._statuses_to_indexed_file_seeds[ file_seed.status ].append( ( index, file_seed ) )
-
+ file_seeds_to_indices[ file_seed ] = index
+ self._FixStatusesToFileSeeds( updated_or_new_file_seeds )
+
self._SetStatusDirty()
- self.NotifyFileSeedsUpdated( updated_or_new_file_seeds )
+ self._NotifyFileSeedsUpdated( updated_or_new_file_seeds )
return len( updated_or_new_file_seeds )
def AdvanceFileSeed( self, file_seed: FileSeed ):
+ updated_file_seeds = []
+
with self._lock:
- if file_seed in self._file_seeds_to_indices:
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
+
+ if file_seed in file_seeds_to_indices:
- index = self._file_seeds_to_indices[ file_seed ]
+ index = file_seeds_to_indices[ file_seed ]
if index > 0:
+ swapped_file_seed = self._file_seeds[ index - 1 ]
+
self._file_seeds.remove( file_seed )
self._file_seeds.insert( index - 1, file_seed )
-
- self._FileSeedIndicesJustChanged()
+ file_seeds_to_indices[ file_seed ] = index - 1
+ file_seeds_to_indices[ swapped_file_seed ] = index
+
+ updated_file_seeds = [ file_seed, swapped_file_seed ]
+
+ self._FixStatusesToFileSeeds( updated_file_seeds )
+
- self.NotifyFileSeedsUpdated( ( file_seed, ) )
+ self._NotifyFileSeedsUpdated( updated_file_seeds )
def CanCompact( self, compact_before_this_source_time: int ):
@@ -2518,49 +2671,54 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
return
- new_file_seeds = HydrusSerialisable.SerialisableList()
+ removee_file_seeds = set()
for file_seed in self._file_seeds[:-self.COMPACT_NUMBER]:
still_to_do = file_seed.status == CC.STATUS_UNKNOWN
still_relevant = self._GetSourceTimestampForVelocityCalculations( file_seed ) > compact_before_this_source_time
- if still_to_do or still_relevant:
+ if not ( still_to_do or still_relevant ):
- new_file_seeds.append( file_seed )
+ removee_file_seeds.add( file_seed )
- new_file_seeds.extend( self._file_seeds[-self.COMPACT_NUMBER:] )
-
- self._file_seeds = new_file_seeds
-
- self._FileSeedIndicesJustChanged()
-
- self._SetStatusDirty()
-
+
+ self.RemoveFileSeeds( removee_file_seeds )
def DelayFileSeed( self, file_seed: FileSeed ):
+ updated_file_seeds = []
+
with self._lock:
- if file_seed in self._file_seeds_to_indices:
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
+
+ if file_seed in file_seeds_to_indices:
- index = self._file_seeds_to_indices[ file_seed ]
+ index = file_seeds_to_indices[ file_seed ]
if index < len( self._file_seeds ) - 1:
+ swapped_file_seed = self._file_seeds[ index + 1 ]
+
self._file_seeds.remove( file_seed )
self._file_seeds.insert( index + 1, file_seed )
-
- self._FileSeedIndicesJustChanged()
+ file_seeds_to_indices[ swapped_file_seed ] = index
+ file_seeds_to_indices[ file_seed ] = index + 1
+
+ updated_file_seeds = [ file_seed, swapped_file_seed ]
+
+ self._FixStatusesToFileSeeds( updated_file_seeds )
+
- self.NotifyFileSeedsUpdated( ( file_seed, ) )
+ self._NotifyFileSeedsUpdated( updated_file_seeds )
def GetAPIInfoDict( self, simple: bool ):
@@ -2656,8 +2814,6 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
def GetFileSeedCount( self, status: int = None ):
- result = 0
-
with self._lock:
if status is None:
@@ -2666,12 +2822,9 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
else:
- if self._statuses_to_indexed_file_seeds_dirty:
-
- self._RegenerateStatusesToFileSeeds()
-
+ statuses_to_file_seeds = self._GetStatusesToFileSeeds()
- return len( self._statuses_to_indexed_file_seeds[ status ] )
+ return len( statuses_to_file_seeds[ status ] )
@@ -2690,7 +2843,9 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
with self._lock:
- return self._file_seeds_to_indices[ file_seed ]
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
+
+ return file_seeds_to_indices[ file_seed ]
@@ -2793,74 +2948,100 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
def InsertFileSeeds( self, index: int, file_seeds: typing.Collection[ FileSeed ] ):
- if len( file_seeds ) == 0:
-
- return 0
-
-
- new_file_seeds = set()
+ file_seeds = HydrusData.DedupeList( file_seeds )
with self._lock:
- index = min( index, len( self._file_seeds ) )
+ new_file_seeds = []
for file_seed in file_seeds:
- if self._HasFileSeed( file_seed ) or file_seed in new_file_seeds:
+ file_seed.Normalise()
+
+ if self._HasFileSeed( file_seed ):
continue
- file_seed.Normalise()
+ new_file_seeds.append( file_seed )
- new_file_seeds.add( file_seed )
+
+ if len( file_seeds ) == 0:
+
+ return 0
+
+
+ index = min( index, len( self._file_seeds ) )
+
+ original_insertion_index = index
+
+ for file_seed in new_file_seeds:
self._file_seeds.insert( index, file_seed )
index += 1
- self._FileSeedIndicesJustChanged()
+ self._SetFileSeedsToIndicesDirty()
self._SetStatusDirty()
+ self._FixStatusesToFileSeeds( new_file_seeds )
+
+ updated_file_seeds = self._file_seeds[ original_insertion_index : ]
+
- self.NotifyFileSeedsUpdated( new_file_seeds )
+ self._NotifyFileSeedsUpdated( updated_file_seeds )
return len( new_file_seeds )
def NotifyFileSeedsUpdated( self, file_seeds: typing.Collection[ FileSeed ] ):
+ if len( file_seeds ) == 0:
+
+ return
+
+
with self._lock:
- if not self._statuses_to_indexed_file_seeds_dirty:
-
- self._FixFileSeedsStatusPosition( file_seeds )
-
-
- #
+ self._FixStatusesToFileSeeds( file_seeds )
self._SetStatusDirty()
- HG.client_controller.pub( 'file_seed_cache_file_seeds_updated', self._file_seed_cache_key, file_seeds )
+ self._NotifyFileSeedsUpdated( file_seeds )
- def RemoveFileSeeds( self, file_seeds: typing.Iterable[ FileSeed ] ):
+ def RemoveFileSeeds( self, file_seeds_to_delete: typing.Iterable[ FileSeed ] ):
with self._lock:
- file_seeds_to_delete = set( file_seeds )
+ file_seeds_to_indices = self._GetFileSeedsToIndices()
+
+ file_seeds_to_delete = { file_seed for file_seed in file_seeds_to_delete if file_seed in file_seeds_to_indices }
+
+ if len( file_seeds_to_delete ) == 0:
+
+ return
+
+
+ earliest_affected_index = min( ( file_seeds_to_indices[ file_seed ] for file_seed in file_seeds_to_delete ) )
self._file_seeds = HydrusSerialisable.SerialisableList( [ file_seed for file_seed in self._file_seeds if file_seed not in file_seeds_to_delete ] )
- self._FileSeedIndicesJustChanged()
+ self._SetFileSeedsToIndicesDirty()
self._SetStatusDirty()
+ self._FixStatusesToFileSeeds( file_seeds_to_delete )
+
+ index_shuffled_file_seeds = self._file_seeds[ earliest_affected_index : ]
+
- self.NotifyFileSeedsUpdated( file_seeds_to_delete )
+ updated_file_seeds = file_seeds_to_delete.union( index_shuffled_file_seeds )
+
+ self._NotifyFileSeedsUpdated( updated_file_seeds )
def RemoveFileSeedsByStatus( self, statuses_to_remove: typing.Collection[ int ] ):
@@ -2894,8 +3075,12 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
file_seed.SetStatus( CC.STATUS_UNKNOWN )
+ self._FixStatusesToFileSeeds( failed_file_seeds )
+
+ self._SetStatusDirty()
+
- self.NotifyFileSeedsUpdated( failed_file_seeds )
+ self._NotifyFileSeedsUpdated( failed_file_seeds )
def RetryIgnored( self, ignored_regex = None ):
@@ -2917,8 +3102,12 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
file_seed.SetStatus( CC.STATUS_UNKNOWN )
+ self._FixStatusesToFileSeeds( ignored_file_seeds )
+
+ self._SetStatusDirty()
+
- self.NotifyFileSeedsUpdated( ignored_file_seeds )
+ self._NotifyFileSeedsUpdated( ignored_file_seeds )
def Reverse( self ):
@@ -2927,10 +3116,12 @@ class FileSeedCache( HydrusSerialisable.SerialisableBase ):
self._file_seeds.reverse()
- self._FileSeedIndicesJustChanged()
+ self._SetFileSeedsToIndicesDirty()
+
+ updated_file_seeds = list( self._file_seeds )
- self.NotifyFileSeedsUpdated( list( self._file_seeds ) )
+ self._NotifyFileSeedsUpdated( updated_file_seeds )
def WorkToDo( self ):
diff --git a/hydrus/client/importing/ClientImportGallerySeeds.py b/hydrus/client/importing/ClientImportGallerySeeds.py
index 372b436d..5fb0a8f5 100644
--- a/hydrus/client/importing/ClientImportGallerySeeds.py
+++ b/hydrus/client/importing/ClientImportGallerySeeds.py
@@ -298,7 +298,7 @@ class GallerySeed( HydrusSerialisable.SerialisableBase ):
return False
- def WorkOnURL( self, gallery_token_name, gallery_seed_log, file_seeds_callable, status_hook, title_hook, network_job_factory, network_job_presentation_context_factory, file_import_options, gallery_urls_seen_before = None ):
+ def WorkOnURL( self, gallery_token_name, gallery_seed_log: "GallerySeedLog", file_seeds_callable, status_hook, title_hook, network_job_factory, network_job_presentation_context_factory, file_import_options, gallery_urls_seen_before = None ):
if gallery_urls_seen_before is None:
@@ -494,7 +494,7 @@ class GallerySeed( HydrusSerialisable.SerialisableBase ):
sub_gallery_seed.SetExternalAdditionalServiceKeysToTags( self._external_additional_service_keys_to_tags )
- gallery_seed_log.AddGallerySeeds( sub_gallery_seeds )
+ gallery_seed_log.AddGallerySeeds( sub_gallery_seeds, parent_gallery_seed = self )
added_new_gallery_pages = True
@@ -569,7 +569,7 @@ class GallerySeed( HydrusSerialisable.SerialisableBase ):
next_gallery_seed.SetExternalAdditionalServiceKeysToTags( self._external_additional_service_keys_to_tags )
- gallery_seed_log.AddGallerySeeds( next_gallery_seeds )
+ gallery_seed_log.AddGallerySeeds( next_gallery_seeds, parent_gallery_seed = self )
added_new_gallery_pages = True
@@ -760,7 +760,7 @@ class GallerySeedLog( HydrusSerialisable.SerialisableBase ):
self._status_dirty = True
- def AddGallerySeeds( self, gallery_seeds ):
+ def AddGallerySeeds( self, gallery_seeds, parent_gallery_seed: typing.Optional[ GallerySeed ] = None ) -> int:
if len( gallery_seeds ) == 0:
@@ -786,23 +786,48 @@ class GallerySeedLog( HydrusSerialisable.SerialisableBase ):
new_gallery_seeds.append( gallery_seed )
- self._gallery_seeds.append( gallery_seed )
-
- self._gallery_seeds_to_indices[ gallery_seed ] = len( self._gallery_seeds ) - 1
-
seen_urls.add( gallery_seed.url )
+ if len( new_gallery_seeds ) == 0:
+
+ return 0
+
+
+ if parent_gallery_seed is None or parent_gallery_seed not in self._gallery_seeds:
+
+ insertion_index = len( self._gallery_seeds )
+
+ else:
+
+ insertion_index = self._gallery_seeds.index( parent_gallery_seed ) + 1
+
+
+ original_insertion_index = insertion_index
+
+ for gallery_seed in new_gallery_seeds:
+
+ self._gallery_seeds.insert( insertion_index, gallery_seed )
+
+ insertion_index += 1
+
+
+ self._gallery_seeds_to_indices = { gallery_seed : index for ( index, gallery_seed ) in enumerate( self._gallery_seeds ) }
+
self._SetStatusDirty()
+ updated_gallery_seeds = self._gallery_seeds[ original_insertion_index : ]
+
- self.NotifyGallerySeedsUpdated( new_gallery_seeds )
+ self.NotifyGallerySeedsUpdated( updated_gallery_seeds )
return len( new_gallery_seeds )
def AdvanceGallerySeed( self, gallery_seed ):
+ updated_gallery_seeds = []
+
with self._lock:
if gallery_seed in self._gallery_seeds_to_indices:
@@ -811,16 +836,20 @@ class GallerySeedLog( HydrusSerialisable.SerialisableBase ):
if index > 0:
+ swapped_gallery_seed = self._gallery_seeds[ index - 1 ]
+
self._gallery_seeds.remove( gallery_seed )
self._gallery_seeds.insert( index - 1, gallery_seed )
-
- self._gallery_seeds_to_indices = { gallery_seed : index for ( index, gallery_seed ) in enumerate( self._gallery_seeds ) }
-
+ self._gallery_seeds_to_indices[ gallery_seed ] = index - 1
+ self._gallery_seeds_to_indices[ swapped_gallery_seed ] = index
+
+ updated_gallery_seeds = ( gallery_seed, swapped_gallery_seed )
+
- self.NotifyGallerySeedsUpdated( ( gallery_seed, ) )
+ self.NotifyGallerySeedsUpdated( updated_gallery_seeds )
def CanCompact( self, compact_before_this_source_time ):
@@ -900,6 +929,8 @@ class GallerySeedLog( HydrusSerialisable.SerialisableBase ):
def DelayGallerySeed( self, gallery_seed ):
+ updated_gallery_seeds = []
+
with self._lock:
if gallery_seed in self._gallery_seeds_to_indices:
@@ -908,16 +939,21 @@ class GallerySeedLog( HydrusSerialisable.SerialisableBase ):
if index < len( self._gallery_seeds ) - 1:
+ swapped_gallery_seed = self._gallery_seeds[ index + 1 ]
+
self._gallery_seeds.remove( gallery_seed )
self._gallery_seeds.insert( index + 1, gallery_seed )
-
- self._gallery_seeds_to_indices = { gallery_seed : index for ( index, gallery_seed ) in enumerate( self._gallery_seeds ) }
+ self._gallery_seeds_to_indices[ swapped_gallery_seed ] = index
+ self._gallery_seeds_to_indices[ gallery_seed ] = index + 1
+
+ updated_gallery_seeds = ( swapped_gallery_seed, gallery_seed )
+
- self.NotifyGallerySeedsUpdated( ( gallery_seed, ) )
+ self.NotifyGallerySeedsUpdated( updated_gallery_seeds )
def GetExampleGallerySeed( self ):
@@ -1062,6 +1098,11 @@ class GallerySeedLog( HydrusSerialisable.SerialisableBase ):
def NotifyGallerySeedsUpdated( self, gallery_seeds ):
+ if len( gallery_seeds ) == 0:
+
+ return
+
+
with self._lock:
self._SetStatusDirty()
@@ -1070,11 +1111,18 @@ class GallerySeedLog( HydrusSerialisable.SerialisableBase ):
HG.client_controller.pub( 'gallery_seed_log_gallery_seeds_updated', self._gallery_seed_log_key, gallery_seeds )
- def RemoveGallerySeeds( self, gallery_seeds ):
+ def RemoveGallerySeeds( self, gallery_seeds_to_delete ):
with self._lock:
- gallery_seeds_to_delete = set( gallery_seeds )
+ gallery_seeds_to_delete = { gallery_seed for gallery_seed in gallery_seeds_to_delete if gallery_seed in self._gallery_seeds_to_indices }
+
+ if len( gallery_seeds_to_delete ) == 0:
+
+ return
+
+
+ earliest_affected_index = min( ( self._gallery_seeds_to_indices[ gallery_seed ] for gallery_seed in gallery_seeds_to_delete ) )
self._gallery_seeds = HydrusSerialisable.SerialisableList( [ gallery_seed for gallery_seed in self._gallery_seeds if gallery_seed not in gallery_seeds_to_delete ] )
@@ -1082,8 +1130,12 @@ class GallerySeedLog( HydrusSerialisable.SerialisableBase ):
self._SetStatusDirty()
+ index_shuffled_gallery_seeds = self._gallery_seeds[ earliest_affected_index : ]
+
- self.NotifyGallerySeedsUpdated( gallery_seeds_to_delete )
+ updated_gallery_seeds = gallery_seeds_to_delete.union( index_shuffled_gallery_seeds )
+
+ self.NotifyGallerySeedsUpdated( updated_gallery_seeds )
def RemoveGallerySeedsByStatus( self, statuses_to_remove ):
diff --git a/hydrus/client/importing/ClientImportLocal.py b/hydrus/client/importing/ClientImportLocal.py
index b9e9657c..df0f3e8a 100644
--- a/hydrus/client/importing/ClientImportLocal.py
+++ b/hydrus/client/importing/ClientImportLocal.py
@@ -669,8 +669,8 @@ class ImportFolder( HydrusSerialisable.SerialisableBaseNamed ):
def _CheckFolder( self, job_key ):
-
- all_paths = ClientFiles.GetAllFilePaths( [ self._path ] )
+
+ ( all_paths, num_sidecars ) = ClientFiles.GetAllFilePaths( [ self._path ] )
all_paths = HydrusPaths.FilterFreePaths( all_paths )
diff --git a/hydrus/client/metadata/ClientMetadataMigrationExporters.py b/hydrus/client/metadata/ClientMetadataMigrationExporters.py
index efa0d746..6eae7d74 100644
--- a/hydrus/client/metadata/ClientMetadataMigrationExporters.py
+++ b/hydrus/client/metadata/ClientMetadataMigrationExporters.py
@@ -130,7 +130,7 @@ class SingleFileMetadataExporterMediaTags( HydrusSerialisable.SerialisableBase,
if len( tags ) > 0:
- content_updates = [ HydrusData.ContentUpdate( HC.CONTENT_TYPE_MAPPINGS, add_content_action, ( tag, hashes ) ) for tag in rows ]
+ content_updates = [ HydrusData.ContentUpdate( HC.CONTENT_TYPE_MAPPINGS, add_content_action, ( tag, hashes ) ) for tag in tags ]
HG.client_controller.WriteSynchronous( 'content_updates', { self._service_key : content_updates } )
diff --git a/hydrus/client/metadata/ClientMetadataMigrationImporters.py b/hydrus/client/metadata/ClientMetadataMigrationImporters.py
index 33850095..263841eb 100644
--- a/hydrus/client/metadata/ClientMetadataMigrationImporters.py
+++ b/hydrus/client/metadata/ClientMetadataMigrationImporters.py
@@ -153,6 +153,9 @@ class SingleFileMetadataImporterMediaTags( HydrusSerialisable.SerialisableBase,
tags = media_result.GetTagsManager().GetCurrent( self._service_key, ClientTags.TAG_DISPLAY_STORAGE )
+ # turning ::) into :)
+ tags = { HydrusText.re_leading_double_colon.sub( ':', tag ) for tag in tags }
+
if self._string_processor.MakesChanges():
tags = self._string_processor.ProcessStrings( tags )
diff --git a/hydrus/client/networking/ClientLocalServer.py b/hydrus/client/networking/ClientLocalServer.py
index 11396995..6e236b08 100644
--- a/hydrus/client/networking/ClientLocalServer.py
+++ b/hydrus/client/networking/ClientLocalServer.py
@@ -46,6 +46,7 @@ class HydrusServiceClientAPI( HydrusClientService ):
root.putChild( b'session_key', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAccountSessionKey( self._service, self._client_requests_domain ) )
root.putChild( b'verify_access_key', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAccountVerify( self._service, self._client_requests_domain ) )
root.putChild( b'get_services', ClientLocalServerResources.HydrusResourceClientAPIRestrictedGetServices( self._service, self._client_requests_domain ) )
+ root.putChild( b'get_service', ClientLocalServerResources.HydrusResourceClientAPIRestrictedGetService( self._service, self._client_requests_domain ) )
add_files = NoResource()
@@ -63,7 +64,6 @@ class HydrusServiceClientAPI( HydrusClientService ):
add_tags.putChild( b'add_tags', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAddTagsAddTags( self._service, self._client_requests_domain ) )
add_tags.putChild( b'clean_tags', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAddTagsCleanTags( self._service, self._client_requests_domain ) )
- add_tags.putChild( b'get_tag_services', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAddTagsGetTagServices( self._service, self._client_requests_domain ) )
add_tags.putChild( b'search_tags', ClientLocalServerResources.HydrusResourceClientAPIRestrictedAddTagsSearchTags( self._service, self._client_requests_domain ) )
add_urls = NoResource()
diff --git a/hydrus/client/networking/ClientLocalServerResources.py b/hydrus/client/networking/ClientLocalServerResources.py
index bff6114f..4c56defa 100644
--- a/hydrus/client/networking/ClientLocalServerResources.py
+++ b/hydrus/client/networking/ClientLocalServerResources.py
@@ -58,12 +58,24 @@ LOCAL_BOORU_JSON_BYTE_LIST_PARAMS = set()
# if a variable name isn't defined here, a GET with it won't work
CLIENT_API_INT_PARAMS = { 'file_id', 'file_sort_type', 'potentials_search_type', 'pixel_duplicates', 'max_hamming_distance', 'max_num_pairs' }
-CLIENT_API_BYTE_PARAMS = { 'hash', 'destination_page_key', 'page_key', 'Hydrus-Client-API-Access-Key', 'Hydrus-Client-API-Session-Key', 'tag_service_key', 'tag_service_key_1', 'tag_service_key_2', 'file_service_key' }
-CLIENT_API_STRING_PARAMS = { 'name', 'url', 'domain', 'search', 'file_service_name', 'tag_service_name', 'reason', 'tag_display_type', 'source_hash_type', 'desired_hash_type' }
-CLIENT_API_JSON_PARAMS = { 'basic_permissions', 'system_inbox', 'system_archive', 'tags', 'tags_1', 'tags_2', 'file_ids', 'only_return_identifiers', 'only_return_basic_information', 'create_new_file_ids', 'detailed_url_information', 'hide_service_names_tags', 'hide_service_keys_tags', 'simple', 'file_sort_asc', 'return_hashes', 'return_file_ids', 'include_notes', 'notes', 'note_names', 'doublecheck_file_system' }
-CLIENT_API_JSON_BYTE_LIST_PARAMS = { 'hashes' }
+CLIENT_API_BYTE_PARAMS = { 'hash', 'destination_page_key', 'page_key', 'service_key', 'Hydrus-Client-API-Access-Key', 'Hydrus-Client-API-Session-Key', 'file_service_key', 'deleted_file_service_key', 'tag_service_key', 'tag_service_key_1', 'tag_service_key_2' }
+CLIENT_API_STRING_PARAMS = { 'name', 'url', 'domain', 'search', 'service_name', 'reason', 'tag_display_type', 'source_hash_type', 'desired_hash_type' }
+CLIENT_API_JSON_PARAMS = { 'basic_permissions', 'tags', 'tags_1', 'tags_2', 'file_ids', 'only_return_identifiers', 'only_return_basic_information', 'create_new_file_ids', 'detailed_url_information', 'hide_service_keys_tags', 'simple', 'file_sort_asc', 'return_hashes', 'return_file_ids', 'include_notes', 'notes', 'note_names', 'doublecheck_file_system' }
+CLIENT_API_JSON_BYTE_LIST_PARAMS = { 'file_service_keys', 'deleted_file_service_keys', 'hashes' }
CLIENT_API_JSON_BYTE_DICT_PARAMS = { 'service_keys_to_tags', 'service_keys_to_actions_to_tags', 'service_keys_to_additional_tags' }
+LEGACY_CLIENT_API_SERVICE_NAME_STRING_PARAMS = { 'file_service_name', 'tag_service_name' }
+CLIENT_API_STRING_PARAMS.update( LEGACY_CLIENT_API_SERVICE_NAME_STRING_PARAMS )
+
+LEGACY_CLIENT_API_SERVICE_NAME_JSON_DICT_PARAMS = { 'service_names_to_tags', 'service_names_to_actions_to_tags', 'service_names_to_additional_tags' }
+CLIENT_API_JSON_PARAMS.update( LEGACY_CLIENT_API_SERVICE_NAME_JSON_DICT_PARAMS )
+
+def ConvertLegacyServiceNameParamToKey( param_name: str ):
+
+ # top tier, works for service_name and service_names
+ return param_name.replace( 'name', 'key' )
+
+
def Dumps( data, mime ):
if mime == HC.APPLICATION_CBOR:
@@ -112,6 +124,25 @@ def CheckHashLength( hashes, hash_type = 'sha256' ):
+def CheckFileService( file_service_key: bytes ):
+
+ try:
+
+ service = HG.client_controller.services_manager.GetService( file_service_key )
+
+ except:
+
+ raise HydrusExceptions.BadRequestException( 'Could not find the file service "{}"!'.format( file_service_key.hex() ) )
+
+
+ if service.GetServiceType() not in HC.ALL_FILE_SERVICES:
+
+ raise HydrusExceptions.BadRequestException( 'Sorry, the service key "{}" did not give a file service!'.format( file_service_key.hex() ) )
+
+
+ return service
+
+
def CheckTagService( tag_service_key: bytes ):
try:
@@ -120,47 +151,93 @@ def CheckTagService( tag_service_key: bytes ):
except:
- raise HydrusExceptions.BadRequestException( 'Could not find that tag service!' )
+ raise HydrusExceptions.BadRequestException( 'Could not find the tag service "{}"!'.format( tag_service_key.hex() ) )
if service.GetServiceType() not in HC.ALL_TAG_SERVICES:
- raise HydrusExceptions.BadRequestException( 'Sorry, that service key did not give a tag service!' )
+ raise HydrusExceptions.BadRequestException( 'Sorry, the service key "{}" did not give a tag service!'.format( tag_service_key.hex() ) )
+ return service
+
-def ConvertServiceNamesDictToKeys( allowed_service_types, service_name_dict ):
+def GetServiceKeyFromName( service_name: str ):
- service_key_dict = {}
-
- for ( service_name, value ) in service_name_dict.items():
+ try:
- try:
-
- service_key = HG.client_controller.services_manager.GetServiceKeyFromName( allowed_service_types, service_name )
-
- except:
-
- raise HydrusExceptions.BadRequestException( 'Could not find the service "{}", or it was the wrong type!'.format( service_name ) )
-
+ service_key = HG.client_controller.services_manager.GetServiceKeyFromName( HC.ALL_SERVICES, service_name )
- service_key_dict[ service_key ] = value
+ except HydrusExceptions.DataMissing:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, did not find a service with name "{}"!'.format( service_name ) )
- return service_key_dict
+ return service_key
+
def ParseLocalBooruGETArgs( requests_args ):
args = HydrusNetworkVariableHandling.ParseTwistedRequestGETArgs( requests_args, LOCAL_BOORU_INT_PARAMS, LOCAL_BOORU_BYTE_PARAMS, LOCAL_BOORU_STRING_PARAMS, LOCAL_BOORU_JSON_PARAMS, LOCAL_BOORU_JSON_BYTE_LIST_PARAMS )
return args
+
+def ParseClientLegacyArgs( args: dict ):
+
+ # adding this v514, so delete when appropriate
+
+ parsed_request_args = HydrusNetworkVariableHandling.ParsedRequestArguments( args )
+
+ legacy_service_string_param_names = LEGACY_CLIENT_API_SERVICE_NAME_STRING_PARAMS.intersection( parsed_request_args.keys() )
+
+ for legacy_service_string_param_name in legacy_service_string_param_names:
+
+ service_name = parsed_request_args[ legacy_service_string_param_name ]
+
+ service_key = GetServiceKeyFromName( service_name )
+
+ del parsed_request_args[ legacy_service_string_param_name ]
+
+ new_service_bytes_param_name = ConvertLegacyServiceNameParamToKey( legacy_service_string_param_name )
+
+ parsed_request_args[ new_service_bytes_param_name ] = service_key
+
+
+ legacy_service_dict_param_names = LEGACY_CLIENT_API_SERVICE_NAME_JSON_DICT_PARAMS.intersection( parsed_request_args.keys() )
+
+ for legacy_service_dict_param_name in legacy_service_dict_param_names:
+
+ service_keys_to_gubbins = {}
+
+ service_names_to_gubbins = parsed_request_args[ legacy_service_dict_param_name ]
+
+ for ( service_name, gubbins ) in service_names_to_gubbins.items():
+
+ service_key = GetServiceKeyFromName( service_name )
+
+ service_keys_to_gubbins[ service_key ] = gubbins
+
+
+ del parsed_request_args[ legacy_service_dict_param_name ]
+
+ new_service_dict_param_name = ConvertLegacyServiceNameParamToKey( legacy_service_dict_param_name )
+
+ parsed_request_args[ new_service_dict_param_name ] = service_keys_to_gubbins
+
+
+ return parsed_request_args
+
+
def ParseClientAPIGETArgs( requests_args ):
args = HydrusNetworkVariableHandling.ParseTwistedRequestGETArgs( requests_args, CLIENT_API_INT_PARAMS, CLIENT_API_BYTE_PARAMS, CLIENT_API_STRING_PARAMS, CLIENT_API_JSON_PARAMS, CLIENT_API_JSON_BYTE_LIST_PARAMS )
+ args = ParseClientLegacyArgs( args )
+
return args
+
def ParseClientAPIPOSTByteArgs( args ):
if not isinstance( args, dict ):
@@ -286,6 +363,8 @@ def ParseClientAPIPOSTByteArgs( args ):
+ parsed_request_args = ParseClientLegacyArgs( parsed_request_args )
+
return parsed_request_args
def ParseClientAPIPOSTArgs( request ):
@@ -342,7 +421,6 @@ def ParseClientAPIPOSTArgs( request ):
parsed_request_args = ParseClientAPIPOSTByteArgs( args )
-
elif request_content_type_mime == HC.APPLICATION_CBOR:
if not CBOR_AVAILABLE:
@@ -357,7 +435,7 @@ def ParseClientAPIPOSTArgs( request ):
args = cbor2.loads( cbor_bytes )
parsed_request_args = ParseClientAPIPOSTByteArgs( args )
-
+
else:
parsed_request_args = HydrusNetworkVariableHandling.ParsedRequestArguments()
@@ -385,8 +463,6 @@ def ParseClientAPISearchPredicates( request ) -> typing.List[ ClientSearch.Predi
default_search_values = {}
default_search_values[ 'tags' ] = []
- default_search_values[ 'system_inbox' ] = False
- default_search_values[ 'system_archive' ] = False
for ( key, value ) in default_search_values.items():
@@ -396,22 +472,10 @@ def ParseClientAPISearchPredicates( request ) -> typing.List[ ClientSearch.Predi
- system_inbox = request.parsed_request_args[ 'system_inbox' ]
- system_archive = request.parsed_request_args[ 'system_archive' ]
-
tags = request.parsed_request_args[ 'tags' ]
predicates = ConvertTagListToPredicates( request, tags )
- if system_inbox:
-
- predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_SYSTEM_INBOX ) )
-
- elif system_archive:
-
- predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_SYSTEM_ARCHIVE ) )
-
-
if len( predicates ) == 0:
return predicates
@@ -436,9 +500,7 @@ def ParseClientAPISearchPredicates( request ) -> typing.List[ ClientSearch.Predi
def ParseDuplicateSearch( request: HydrusServerRequest.HydrusRequest ):
- # TODO: When we have ParseLocationContext for clever file searching, swap it in here too
- # LocationContext has to be the same for both searches
- location_context = ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY )
+ location_context = ParseLocationContext( request, ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY ) )
tag_service_key_1 = request.parsed_request_args.GetValue( 'tag_service_key_1', bytes, default_value = CC.COMBINED_TAG_SERVICE_KEY )
tag_service_key_2 = request.parsed_request_args.GetValue( 'tag_service_key_2', bytes, default_value = CC.COMBINED_TAG_SERVICE_KEY )
@@ -487,43 +549,55 @@ def ParseDuplicateSearch( request: HydrusServerRequest.HydrusRequest ):
)
-def ParseLocationContext( request: HydrusServerRequest.HydrusRequest, default: ClientLocation.LocationContext ):
+def ParseLocationContext( request: HydrusServerRequest.HydrusRequest, default: ClientLocation.LocationContext, deleted_allowed = True ):
- if 'file_service_key' in request.parsed_request_args or 'file_service_name' in request.parsed_request_args:
+ current_file_service_keys = set()
+ deleted_file_service_keys = set()
+
+ if 'file_service_key' in request.parsed_request_args:
- if 'file_service_key' in request.parsed_request_args:
+ file_service_key = request.parsed_request_args.GetValue( 'file_service_key', bytes )
+
+ current_file_service_keys.add( file_service_key )
+
+
+ if 'file_service_keys' in request.parsed_request_args:
+
+ file_service_keys = request.parsed_request_args.GetValue( 'file_service_keys', list, expected_list_type = bytes )
+
+ current_file_service_keys.update( file_service_keys )
+
+
+ if deleted_allowed:
+
+ if 'deleted_file_service_key' in request.parsed_request_args:
- file_service_key = request.parsed_request_args[ 'file_service_key' ]
+ file_service_key = request.parsed_request_args.GetValue( 'deleted_file_service_key', bytes )
- else:
-
- file_service_name = request.parsed_request_args[ 'file_service_name' ]
-
- try:
-
- file_service_key = HG.client_controller.services_manager.GetServiceKeyFromName( HC.ALL_FILE_SERVICES, file_service_name )
-
- except:
-
- raise HydrusExceptions.BadRequestException( 'Could not find the service "{}"!'.format( file_service_name ) )
-
+ deleted_file_service_keys.add( file_service_key )
- try:
+ if 'deleted_file_service_keys' in request.parsed_request_args:
- service_type = HG.client_controller.services_manager.GetServiceType( file_service_key )
+ file_service_keys = request.parsed_request_args.GetValue( 'deleted_file_service_keys', list, expected_list_type = bytes )
- except:
-
- raise HydrusExceptions.BadRequestException( 'Could not find that file service!' )
+ deleted_file_service_keys.update( file_service_keys )
- if service_type not in HC.ALL_FILE_SERVICES:
-
- raise HydrusExceptions.BadRequestException( 'Sorry, that service key did not give a file service!' )
-
+
+ for service_key in current_file_service_keys:
- return ClientLocation.LocationContext.STATICCreateSimple( file_service_key )
+ CheckFileService( service_key )
+
+
+ for service_key in deleted_file_service_keys:
+
+ CheckFileService( service_key )
+
+
+ if len( current_file_service_keys ) > 0 or len( deleted_file_service_keys ) > 0:
+
+ return ClientLocation.LocationContext( current_service_keys = current_file_service_keys, deleted_service_keys = deleted_file_service_keys )
else:
@@ -533,40 +607,60 @@ def ParseLocationContext( request: HydrusServerRequest.HydrusRequest, default: C
def ParseHashes( request: HydrusServerRequest.HydrusRequest ):
- hashes = set()
+ something_was_set = False
+
+ hashes = []
if 'hash' in request.parsed_request_args:
+ something_was_set = True
+
hash = request.parsed_request_args.GetValue( 'hash', bytes )
- hashes.add( hash )
+ hashes.append( hash )
if 'hashes' in request.parsed_request_args:
+ something_was_set = True
+
more_hashes = request.parsed_request_args.GetValue( 'hashes', list, expected_list_type = bytes )
- hashes.update( more_hashes )
+ hashes.extend( more_hashes )
if 'file_id' in request.parsed_request_args:
+ something_was_set = True
+
hash_id = request.parsed_request_args.GetValue( 'file_id', int )
hash_ids_to_hashes = HG.client_controller.Read( 'hash_ids_to_hashes', hash_ids = [ hash_id ] )
- hashes.update( hash_ids_to_hashes.values() )
+ if len( hash_ids_to_hashes ) > 0:
+
+ hashes.extend( hash_ids_to_hashes[ hash_id ] )
+
if 'file_ids' in request.parsed_request_args:
+ something_was_set = True
+
hash_ids = request.parsed_request_args.GetValue( 'file_ids', list, expected_list_type = int )
hash_ids_to_hashes = HG.client_controller.Read( 'hash_ids_to_hashes', hash_ids = hash_ids )
- hashes.update( hash_ids_to_hashes.values() )
+ hashes.extend( [ hash_ids_to_hashes[ hash_id ] for hash_id in hash_ids ] )
+ if not something_was_set: # subtly different to 'no hashes'
+
+ raise HydrusExceptions.BadRequestException( 'Please include some files in your request--file_id or hash based!' )
+
+
+ hashes = HydrusData.DedupeList( hashes )
+
CheckHashLength( hashes )
return hashes
@@ -619,25 +713,12 @@ def ParseRequestedResponseMime( request: HydrusServerRequest.HydrusRequest ):
def ParseTagServiceKey( request: HydrusServerRequest.HydrusRequest ):
- if 'tag_service_key' in request.parsed_request_args or 'tag_service_name' in request.parsed_request_args:
+ if 'tag_service_key' in request.parsed_request_args:
if 'tag_service_key' in request.parsed_request_args:
tag_service_key = request.parsed_request_args[ 'tag_service_key' ]
- else:
-
- tag_service_name = request.parsed_request_args[ 'tag_service_name' ]
-
- try:
-
- tag_service_key = HG.client_controller.services_manager.GetServiceKeyFromName( HC.ALL_TAG_SERVICES, tag_service_name )
-
- except:
-
- raise HydrusExceptions.BadRequestException( 'Could not find the service "{}"!'.format( tag_service_name ) )
-
-
CheckTagService( tag_service_key )
@@ -1344,6 +1425,90 @@ class HydrusResourceClientAPIRestrictedAccountVerify( HydrusResourceClientAPIRes
return response_context
+class HydrusResourceClientAPIRestrictedGetService( HydrusResourceClientAPIRestricted ):
+
+ def _CheckAPIPermissions( self, request: HydrusServerRequest.HydrusRequest ):
+
+ request.client_api_permissions.CheckAtLeastOnePermission(
+ (
+ ClientAPI.CLIENT_API_PERMISSION_ADD_FILES,
+ ClientAPI.CLIENT_API_PERMISSION_ADD_TAGS,
+ ClientAPI.CLIENT_API_PERMISSION_ADD_NOTES,
+ ClientAPI.CLIENT_API_PERMISSION_MANAGE_PAGES,
+ ClientAPI.CLIENT_API_PERMISSION_MANAGE_FILE_RELATIONSHIPS,
+ ClientAPI.CLIENT_API_PERMISSION_SEARCH_FILES
+ )
+ )
+
+
+ def _threadDoGETJob( self, request: HydrusServerRequest.HydrusRequest ):
+
+ allowed_service_types = {
+ HC.LOCAL_TAG,
+ HC.TAG_REPOSITORY,
+ HC.LOCAL_FILE_DOMAIN,
+ HC.LOCAL_FILE_UPDATE_DOMAIN,
+ HC.FILE_REPOSITORY,
+ HC.COMBINED_LOCAL_FILE,
+ HC.COMBINED_LOCAL_MEDIA,
+ HC.COMBINED_FILE,
+ HC.COMBINED_TAG,
+ HC.LOCAL_FILE_TRASH_DOMAIN
+ }
+
+ if 'service_key' in request.parsed_request_args:
+
+ service_key = request.parsed_request_args.GetValue( 'service_key', bytes )
+
+ elif 'service_name' in request.parsed_request_args:
+
+ service_name = request.parsed_request_args.GetValue( 'service_name', str )
+
+ try:
+
+ service_key = HG.client_controller.services_manager.GetServiceKeyFromName( allowed_service_types, service_name )
+
+ except HydrusExceptions.DataMissing:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, did not find a service with name "{}"!'.format( service_name ) )
+
+
+ else:
+
+ raise HydrusExceptions.BadRequestException( 'Sorry, you need to give a service_key or service_name!' )
+
+
+ try:
+
+ service = HG.client_controller.services_manager.GetService( service_key )
+
+ except HydrusExceptions.DataMissing:
+
+ raise HydrusExceptions.NotFoundException( 'Sorry, did not find a service with key "{}"!'.format( service_key.hex() ) )
+
+
+ if service.GetServiceType() not in allowed_service_types:
+
+ raise HydrusExceptions.BadRequestException( 'Sorry, for now, you cannot ask about this service!' )
+
+
+ body_dict = {
+ 'service' : {
+ 'name' : service.GetName(),
+ 'type' : service.GetServiceType(),
+ 'type_pretty' : HC.service_string_lookup[ service.GetServiceType() ],
+ 'service_key' : service.GetServiceKey().hex()
+ }
+ }
+
+ body = Dumps( body_dict, request.preferred_mime )
+
+ response_context = HydrusServerResources.ResponseContext( 200, mime = request.preferred_mime, body = body )
+
+ return response_context
+
+
+
class HydrusResourceClientAPIRestrictedGetServices( HydrusResourceClientAPIRestricted ):
def _CheckAPIPermissions( self, request: HydrusServerRequest.HydrusRequest ):
@@ -1464,7 +1629,7 @@ class HydrusResourceClientAPIRestrictedAddFilesArchiveFiles( HydrusResourceClien
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
- hashes = ParseHashes( request )
+ hashes = set( ParseHashes( request ) )
content_update = HydrusData.ContentUpdate( HC.CONTENT_TYPE_FILES, HC.CONTENT_UPDATE_ARCHIVE, hashes )
@@ -1484,7 +1649,7 @@ class HydrusResourceClientAPIRestrictedAddFilesDeleteFiles( HydrusResourceClient
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
- location_context = ParseLocationContext( request, ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY ) )
+ location_context = ParseLocationContext( request, ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY ), deleted_allowed = False )
if 'reason' in request.parsed_request_args:
@@ -1495,7 +1660,7 @@ class HydrusResourceClientAPIRestrictedAddFilesDeleteFiles( HydrusResourceClient
reason = 'Deleted via Client API.'
- hashes = ParseHashes( request )
+ hashes = set( ParseHashes( request ) )
# expand this to take reason
@@ -1519,7 +1684,7 @@ class HydrusResourceClientAPIRestrictedAddFilesUnarchiveFiles( HydrusResourceCli
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
- hashes = ParseHashes( request )
+ hashes = set( ParseHashes( request ) )
content_update = HydrusData.ContentUpdate( HC.CONTENT_TYPE_FILES, HC.CONTENT_UPDATE_INBOX, hashes )
@@ -1538,7 +1703,7 @@ class HydrusResourceClientAPIRestrictedAddFilesUndeleteFiles( HydrusResourceClie
location_context = ParseLocationContext( request, ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY ) )
- hashes = ParseHashes( request )
+ hashes = set( ParseHashes( request ) )
location_context.LimitToServiceTypes( HG.client_controller.services_manager.GetServiceType, ( HC.LOCAL_FILE_DOMAIN, HC.COMBINED_LOCAL_MEDIA ) )
@@ -1677,38 +1842,31 @@ class HydrusResourceClientAPIRestrictedAddTagsAddTags( HydrusResourceClientAPIRe
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
- hashes = ParseHashes( request )
+ hashes = set( ParseHashes( request ) )
#
service_keys_to_tags = None
+ service_keys_to_actions_to_tags = None
+
if 'service_keys_to_tags' in request.parsed_request_args:
service_keys_to_tags = request.parsed_request_args.GetValue( 'service_keys_to_tags', dict )
- elif 'service_names_to_tags' in request.parsed_request_args:
-
- service_names_to_tags = request.parsed_request_args.GetValue( 'service_names_to_tags', dict )
-
- service_keys_to_tags = ConvertServiceNamesDictToKeys( HC.REAL_TAG_SERVICES, service_names_to_tags )
-
-
- service_keys_to_actions_to_tags = None
-
- if service_keys_to_tags is not None:
-
service_keys_to_actions_to_tags = {}
for ( service_key, tags ) in service_keys_to_tags.items():
- try:
+ service = CheckTagService( service_key )
+
+ HydrusNetworkVariableHandling.TestVariableType( 'tags in service_keys_to_tags', tags, list, expected_list_type = str )
+
+ tags = HydrusTags.CleanTags( tags )
+
+ if len( tags ) == 0:
- service = HG.client_controller.services_manager.GetService( service_key )
-
- except:
-
- raise HydrusExceptions.BadRequestException( 'Could not find the service with key {}! Maybe it was recently deleted?'.format( service_key.hex() ) )
+ continue
if service.GetServiceType() == HC.LOCAL_TAG:
@@ -1728,13 +1886,58 @@ class HydrusResourceClientAPIRestrictedAddTagsAddTags( HydrusResourceClientAPIRe
if 'service_keys_to_actions_to_tags' in request.parsed_request_args:
- service_keys_to_actions_to_tags = request.parsed_request_args.GetValue( 'service_keys_to_actions_to_tags', dict )
+ parsed_service_keys_to_actions_to_tags = request.parsed_request_args.GetValue( 'service_keys_to_actions_to_tags', dict )
- elif 'service_names_to_actions_to_tags' in request.parsed_request_args:
+ service_keys_to_actions_to_tags = {}
- service_names_to_actions_to_tags = request.parsed_request_args.GetValue( 'service_names_to_actions_to_tags', dict )
-
- service_keys_to_actions_to_tags = ConvertServiceNamesDictToKeys( HC.REAL_TAG_SERVICES, service_names_to_actions_to_tags )
+ for ( service_key, parsed_actions_to_tags ) in parsed_service_keys_to_actions_to_tags.items():
+
+ service = CheckTagService( service_key )
+
+ HydrusNetworkVariableHandling.TestVariableType( 'actions_to_tags', parsed_actions_to_tags, dict )
+
+ actions_to_tags = {}
+
+ for ( parsed_content_action, tags ) in parsed_actions_to_tags.items():
+
+ HydrusNetworkVariableHandling.TestVariableType( 'action in actions_to_tags', parsed_content_action, str )
+
+ try:
+
+ content_action = int( parsed_content_action )
+
+ except:
+
+ raise HydrusExceptions.BadRequestException( 'Sorry, got an action, "{}", that was not an integer!'.format( parsed_content_action ) )
+
+
+ if service.GetServiceType() == HC.LOCAL_TAG:
+
+ if content_action not in ( HC.CONTENT_UPDATE_ADD, HC.CONTENT_UPDATE_DELETE ):
+
+ raise HydrusExceptions.BadRequestException( 'Sorry, you submitted a content action of "{}" for service "{}", but you can only add/delete on a local tag service!'.format( parsed_content_action, service_key.hex() ) )
+
+
+ else:
+
+ if content_action in ( HC.CONTENT_UPDATE_ADD, HC.CONTENT_UPDATE_DELETE ):
+
+ raise HydrusExceptions.BadRequestException( 'Sorry, you submitted a content action of "{}" for service "{}", but you cannot add/delete on a remote tag service!'.format( parsed_content_action, service_key.hex() ) )
+
+
+
+ HydrusNetworkVariableHandling.TestVariableType( 'tags in actions_to_tags', tags, list ) # do not test for str here, it can be reason tuples!
+
+ actions_to_tags[ content_action ] = tags
+
+
+ if len( actions_to_tags ) == 0:
+
+ continue
+
+
+ service_keys_to_actions_to_tags[ service_key ] = actions_to_tags
+
if service_keys_to_actions_to_tags is None:
@@ -1746,32 +1949,13 @@ class HydrusResourceClientAPIRestrictedAddTagsAddTags( HydrusResourceClientAPIRe
for ( service_key, actions_to_tags ) in service_keys_to_actions_to_tags.items():
- try:
-
- service = HG.client_controller.services_manager.GetService( service_key )
-
- except HydrusExceptions.DataMissing:
-
- raise HydrusExceptions.BadRequestException( 'Could not find the service with key {}! Maybe it was recently deleted?'.format( service_key.hex() ) )
-
-
- if service.GetServiceType() not in HC.REAL_TAG_SERVICES:
-
- raise HydrusExceptions.BadRequestException( 'Was given a service that is not a tag service!' )
-
-
for ( content_action, tags ) in actions_to_tags.items():
tags = list( tags )
- if len( tags ) == 0:
-
- continue
-
-
content_action = int( content_action )
- actual_tags = []
+ content_update_tags = []
tags_to_reasons = {}
@@ -1797,41 +1981,31 @@ class HydrusResourceClientAPIRestrictedAddTagsAddTags( HydrusResourceClientAPIRe
continue
- actual_tags.append( tag )
+ try:
+
+ tag = HydrusTags.CleanTag( tag )
+
+ except:
+
+ continue
+
+
+ content_update_tags.append( tag )
tags_to_reasons[ tag ] = reason
- actual_tags = HydrusTags.CleanTags( actual_tags )
-
- if len( actual_tags ) == 0:
+ if len( content_update_tags ) == 0:
continue
- tags = actual_tags
-
- if service.GetServiceType() == HC.LOCAL_TAG:
-
- if content_action not in ( HC.CONTENT_UPDATE_ADD, HC.CONTENT_UPDATE_DELETE ):
-
- continue
-
-
- else:
-
- if content_action in ( HC.CONTENT_UPDATE_ADD, HC.CONTENT_UPDATE_DELETE ):
-
- continue
-
-
-
if content_action == HC.CONTENT_UPDATE_PETITION:
- content_updates = [ HydrusData.ContentUpdate( HC.CONTENT_TYPE_MAPPINGS, content_action, ( tag, hashes ), reason = tags_to_reasons[ tag ] ) for tag in tags ]
+ content_updates = [ HydrusData.ContentUpdate( HC.CONTENT_TYPE_MAPPINGS, content_action, ( tag, hashes ), reason = tags_to_reasons[ tag ] ) for tag in content_update_tags ]
else:
- content_updates = [ HydrusData.ContentUpdate( HC.CONTENT_TYPE_MAPPINGS, content_action, ( tag, hashes ) ) for tag in tags ]
+ content_updates = [ HydrusData.ContentUpdate( HC.CONTENT_TYPE_MAPPINGS, content_action, ( tag, hashes ) ) for tag in content_update_tags ]
service_keys_to_content_updates[ service_key ].extend( content_updates )
@@ -1848,25 +2022,7 @@ class HydrusResourceClientAPIRestrictedAddTagsAddTags( HydrusResourceClientAPIRe
return response_context
-class HydrusResourceClientAPIRestrictedAddTagsGetTagServices( HydrusResourceClientAPIRestrictedAddTags ):
-
- def _threadDoGETJob( self, request: HydrusServerRequest.HydrusRequest ):
-
- local_tags = HG.client_controller.services_manager.GetServices( ( HC.LOCAL_TAG, ) )
- tag_repos = HG.client_controller.services_manager.GetServices( ( HC.TAG_REPOSITORY, ) )
-
- body_dict = {}
-
- body_dict[ 'local_tags' ] = [ service.GetName() for service in local_tags ]
- body_dict[ 'tag_repositories' ] = [ service.GetName() for service in tag_repos ]
-
- body = Dumps( body_dict, request.preferred_mime )
-
- response_context = HydrusServerResources.ResponseContext( 200, mime = request.preferred_mime, body = body )
-
- return response_context
-
-
+
class HydrusResourceClientAPIRestrictedAddTagsSearchTags( HydrusResourceClientAPIRestrictedAddTags ):
def _CheckAPIPermissions( self, request: HydrusServerRequest.HydrusRequest ):
@@ -1897,9 +2053,9 @@ class HydrusResourceClientAPIRestrictedAddTagsSearchTags( HydrusResourceClientAP
autocomplete_search_text = parsed_autocomplete_text.GetSearchText( True )
- default_location_context = HG.client_controller.new_options.GetDefaultLocalLocationContext()
+ location_context = ParseLocationContext( request, ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY ) )
- file_search_context = ClientSearch.FileSearchContext( location_context = default_location_context, tag_context = tag_context )
+ file_search_context = ClientSearch.FileSearchContext( location_context = location_context, tag_context = tag_context )
job_key = ClientThreading.JobKey( cancellable = True )
@@ -1995,15 +2151,7 @@ class HydrusResourceClientAPIRestrictedAddURLsAssociateURL( HydrusResourceClient
urls = request.parsed_request_args.GetValue( 'urls_to_add', list, expected_list_type = str )
- for url in urls:
-
- if not isinstance( url, str ):
-
- continue
-
-
- urls_to_add.append( url )
-
+ urls_to_add.extend( urls )
urls_to_delete = []
@@ -2021,12 +2169,7 @@ class HydrusResourceClientAPIRestrictedAddURLsAssociateURL( HydrusResourceClient
for url in urls:
- if not isinstance( url, str ):
-
- continue
-
-
- urls_to_delete.append( url )
+ urls_to_delete.extend( urls )
@@ -2046,7 +2189,7 @@ class HydrusResourceClientAPIRestrictedAddURLsAssociateURL( HydrusResourceClient
raise HydrusExceptions.BadRequestException( 'Did not find any URLs to add or delete!' )
- applicable_hashes = ParseHashes( request )
+ applicable_hashes = set( ParseHashes( request ) )
if len( applicable_hashes ) == 0:
@@ -2187,38 +2330,15 @@ class HydrusResourceClientAPIRestrictedAddURLsImportURL( HydrusResourceClientAPI
additional_service_keys_to_tags = ClientTags.ServiceKeysToTags()
- service_keys_to_additional_tags = None
-
- if 'service_names_to_tags' in request.parsed_request_args or 'service_names_to_additional_tags' in request.parsed_request_args:
-
- if 'service_names_to_tags' in request.parsed_request_args:
-
- service_names_to_additional_tags = request.parsed_request_args.GetValue( 'service_names_to_tags', dict )
-
- else:
-
- service_names_to_additional_tags = request.parsed_request_args.GetValue( 'service_names_to_additional_tags', dict )
-
-
- service_keys_to_additional_tags = ConvertServiceNamesDictToKeys( HC.REAL_TAG_SERVICES, service_names_to_additional_tags )
-
- elif 'service_keys_to_additional_tags' in request.parsed_request_args:
+ if 'service_keys_to_additional_tags' in request.parsed_request_args:
service_keys_to_additional_tags = request.parsed_request_args.GetValue( 'service_keys_to_additional_tags', dict )
-
- if service_keys_to_additional_tags is not None:
-
request.client_api_permissions.CheckPermission( ClientAPI.CLIENT_API_PERMISSION_ADD_TAGS )
for ( service_key, tags ) in service_keys_to_additional_tags.items():
- service = HG.client_controller.services_manager.GetService( service_key )
-
- if service.GetServiceType() not in HC.REAL_TAG_SERVICES:
-
- raise HydrusExceptions.BadRequestException( 'Was given a service that is not a tag service!' )
-
+ CheckTagService( service_key )
tags = HydrusTags.CleanTags( tags )
@@ -2487,78 +2607,22 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
def _threadDoGETJob( self, request: HydrusServerRequest.HydrusRequest ):
- missing_hashes = set()
-
only_return_identifiers = request.parsed_request_args.GetValue( 'only_return_identifiers', bool, default_value = False )
only_return_basic_information = request.parsed_request_args.GetValue( 'only_return_basic_information', bool, default_value = False )
- hide_service_names_tags = request.parsed_request_args.GetValue( 'hide_service_names_tags', bool, default_value = True )
- hide_service_keys_tags = request.parsed_request_args.GetValue( 'hide_service_names_tags', bool, default_value = False )
+ hide_service_keys_tags = request.parsed_request_args.GetValue( 'hide_service_keys_tags', bool, default_value = True )
detailed_url_information = request.parsed_request_args.GetValue( 'detailed_url_information', bool, default_value = False )
include_notes = request.parsed_request_args.GetValue( 'include_notes', bool, default_value = False )
create_new_file_ids = request.parsed_request_args.GetValue( 'create_new_file_ids', bool, default_value = False )
- if 'file_ids' in request.parsed_request_args or 'file_id' in request.parsed_request_args:
-
- if 'file_ids' in request.parsed_request_args:
-
- file_ids = request.parsed_request_args.GetValue( 'file_ids', list, expected_list_type = int )
-
- else:
-
- file_ids = [ request.parsed_request_args.GetValue( 'file_id', int ) ]
-
- request.client_api_permissions.CheckPermissionToSeeFiles( file_ids )
-
- elif 'hashes' in request.parsed_request_args or 'hash' in request.parsed_request_args:
-
- request.client_api_permissions.CheckCanSeeAllFiles()
-
- if 'hashes' in request.parsed_request_args:
-
- hashes = request.parsed_request_args.GetValue( 'hashes', list, expected_list_type = bytes )
-
- else:
-
- hashes = [ request.parsed_request_args.GetValue( 'hash', bytes ) ]
-
-
- hashes = HydrusData.DedupeList( hashes )
-
- CheckHashLength( hashes )
-
- file_ids_to_hashes = HG.client_controller.Read( 'hash_ids_to_hashes', hashes = hashes, create_new_hash_ids = create_new_file_ids )
-
- file_ids = set( file_ids_to_hashes.keys() )
-
- if len( file_ids_to_hashes ) < len( hashes ):
-
- missing_hashes = set( hashes ).difference( file_ids_to_hashes.values() )
-
-
- else:
-
- raise HydrusExceptions.BadRequestException( 'Please include a file_ids or hashes parameter!' )
-
+ hashes = ParseHashes( request )
- try:
-
- if only_return_identifiers:
-
- file_ids_to_hashes = HG.client_controller.Read( 'hash_ids_to_hashes', hash_ids = file_ids )
-
- elif only_return_basic_information:
-
- file_info_managers = HG.client_controller.Read( 'file_info_managers_from_ids', file_ids, sorted = True )
-
- else:
-
- media_results = HG.client_controller.Read( 'media_results_from_ids', file_ids, sorted = True )
-
-
- except HydrusExceptions.DataMissing as e:
-
- raise HydrusExceptions.NotFoundException( 'One or more of those file identifiers did not exist in the database!' )
-
+ file_ids_to_hashes = HG.client_controller.Read( 'hash_ids_to_hashes', hashes = hashes, create_new_hash_ids = create_new_file_ids )
+
+ missing_hashes = set( hashes ).difference( file_ids_to_hashes.values() )
+
+ file_ids = set( file_ids_to_hashes.keys() )
+
+ request.client_api_permissions.CheckPermissionToSeeFiles( file_ids )
body_dict = {}
@@ -2588,6 +2652,8 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
elif only_return_basic_information:
+ file_info_managers = HG.client_controller.Read( 'file_info_managers_from_ids', file_ids, sorted = True )
+
for file_info_manager in file_info_managers:
metadata_row = {
@@ -2609,6 +2675,8 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
else:
+ media_results = HG.client_controller.Read( 'media_results_from_ids', file_ids, sorted = True )
+
services_manager = HG.client_controller.services_manager
tag_service_keys = services_manager.GetServiceKeys( HC.ALL_TAG_SERVICES )
@@ -2783,7 +2851,6 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
# Old stuff starts here
- service_names_to_statuses_to_tags = {}
api_service_keys_to_statuses_to_tags = {}
service_keys_to_statuses_to_tags = tags_manager.GetServiceKeysToStatusesToTags( ClientTags.TAG_DISPLAY_STORAGE )
@@ -2794,19 +2861,10 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
if len( statuses_to_tags_json_serialisable ) > 0:
- service_name = service_keys_to_names[ service_key ]
-
- service_names_to_statuses_to_tags[ service_name ] = statuses_to_tags_json_serialisable
-
api_service_keys_to_statuses_to_tags[ service_key.hex() ] = statuses_to_tags_json_serialisable
- if not hide_service_names_tags:
-
- metadata_row[ 'service_names_to_statuses_to_tags' ] = service_names_to_statuses_to_tags
-
-
if not hide_service_keys_tags:
metadata_row[ 'service_keys_to_statuses_to_tags' ] = api_service_keys_to_statuses_to_tags
@@ -2814,7 +2872,6 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
#
- service_names_to_statuses_to_tags = {}
api_service_keys_to_statuses_to_tags = {}
service_keys_to_statuses_to_tags = tags_manager.GetServiceKeysToStatusesToTags( ClientTags.TAG_DISPLAY_ACTUAL )
@@ -2825,19 +2882,10 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
if len( statuses_to_tags_json_serialisable ) > 0:
- service_name = service_keys_to_names[ service_key ]
-
- service_names_to_statuses_to_tags[ service_name ] = statuses_to_tags_json_serialisable
-
api_service_keys_to_statuses_to_tags[ service_key.hex() ] = statuses_to_tags_json_serialisable
- if not hide_service_names_tags:
-
- metadata_row[ 'service_names_to_statuses_to_display_tags' ] = service_names_to_statuses_to_tags
-
-
if not hide_service_keys_tags:
metadata_row[ 'service_keys_to_statuses_to_display_tags' ] = api_service_keys_to_statuses_to_tags
@@ -2861,6 +2909,7 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
return response_context
+
class HydrusResourceClientAPIRestrictedGetFilesGetThumbnail( HydrusResourceClientAPIRestrictedGetFiles ):
def _threadDoGETJob( self, request: HydrusServerRequest.HydrusRequest ):
@@ -3147,8 +3196,7 @@ class HydrusResourceClientAPIRestrictedManageFileRelationshipsGetRelationships(
def _threadDoGETJob( self, request: HydrusServerRequest.HydrusRequest ):
- # TODO: When we have ParseLocationContext for clever file searching, swap it in here too
- location_context = ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY )
+ location_context = ParseLocationContext( request, ClientLocation.LocationContext.STATICCreateSimple( CC.COMBINED_LOCAL_MEDIA_SERVICE_KEY ) )
hashes = ParseHashes( request )
@@ -3262,21 +3310,59 @@ class HydrusResourceClientAPIRestrictedManageFileRelationshipsSetRelationships(
def _threadDoPOSTJob( self, request: HydrusServerRequest.HydrusRequest ):
- rows = []
+ database_write_rows = []
- raw_rows = request.parsed_request_args.GetValue( 'pair_rows', list, expected_list_type = list )
+ raw_rows = []
all_hashes = set()
- for row in raw_rows:
+ pair_rows_old_arg_raw_rows = request.parsed_request_args.GetValue( 'pair_rows', list, expected_list_type = list, default_value = [] )
+
+ for row in pair_rows_old_arg_raw_rows:
if len( row ) != 6:
raise HydrusExceptions.BadRequestException( 'One of the pair rows was the wrong length!' )
+ raw_rows.append( row )
+
+
+ raw_relationship_dicts = request.parsed_request_args.GetValue( 'relationships', list, expected_list_type = dict, default_value = [] )
+
+ for raw_relationship_dict in raw_relationship_dicts:
+
+ duplicate_type = HydrusNetworkVariableHandling.GetValueFromDict( raw_relationship_dict, 'relationship', int )
+ hash_a_hex = HydrusNetworkVariableHandling.GetValueFromDict( raw_relationship_dict, 'hash_a', str )
+ hash_b_hex = HydrusNetworkVariableHandling.GetValueFromDict( raw_relationship_dict, 'hash_b', str )
+ do_default_content_merge = HydrusNetworkVariableHandling.GetValueFromDict( raw_relationship_dict, 'do_default_content_merge', bool )
+ delete_a = HydrusNetworkVariableHandling.GetValueFromDict( raw_relationship_dict, 'delete_a', bool, default_value = False )
+ delete_b = HydrusNetworkVariableHandling.GetValueFromDict( raw_relationship_dict, 'delete_b', bool, default_value = False )
+
+ raw_rows.append( ( duplicate_type, hash_a_hex, hash_b_hex, do_default_content_merge, delete_a, delete_b ) )
+
+
+ allowed_duplicate_types = {
+ HC.DUPLICATE_FALSE_POSITIVE,
+ HC.DUPLICATE_ALTERNATE,
+ HC.DUPLICATE_BETTER,
+ HC.DUPLICATE_WORSE,
+ HC.DUPLICATE_SAME_QUALITY,
+ HC.DUPLICATE_POTENTIAL
+ }
+
+ # variable type testing
+ for row in raw_rows:
+
( duplicate_type, hash_a_hex, hash_b_hex, do_default_content_merge, delete_first, delete_second ) = row
+ HydrusNetworkVariableHandling.TestVariableType( 'relationship', duplicate_type, int, allowed_values = allowed_duplicate_types )
+ HydrusNetworkVariableHandling.TestVariableType( 'hash_a', hash_a_hex, str )
+ HydrusNetworkVariableHandling.TestVariableType( 'hash_b', hash_b_hex, str )
+ HydrusNetworkVariableHandling.TestVariableType( 'do_default_content_merge', do_default_content_merge, bool )
+ HydrusNetworkVariableHandling.TestVariableType( 'delete_first', delete_first, bool )
+ HydrusNetworkVariableHandling.TestVariableType( 'delete_second', delete_second, bool )
+
try:
hash_a = bytes.fromhex( hash_a_hex )
@@ -3300,44 +3386,8 @@ class HydrusResourceClientAPIRestrictedManageFileRelationshipsSetRelationships(
( duplicate_type, hash_a_hex, hash_b_hex, do_default_content_merge, delete_first, delete_second ) = row
- if duplicate_type not in [
- HC.DUPLICATE_FALSE_POSITIVE,
- HC.DUPLICATE_ALTERNATE,
- HC.DUPLICATE_BETTER,
- HC.DUPLICATE_WORSE,
- HC.DUPLICATE_SAME_QUALITY,
- HC.DUPLICATE_POTENTIAL
- ]:
-
- raise HydrusExceptions.BadRequestException( 'One of the duplicate statuses ({}) was incorrect!'.format( duplicate_type ) )
-
-
- try:
-
- hash_a = bytes.fromhex( hash_a_hex )
- hash_b = bytes.fromhex( hash_b_hex )
-
- except:
-
- raise HydrusExceptions.BadRequestException( 'Sorry, did not understand one of the hashes {} or {}!'.format( hash_a_hex, hash_b_hex ) )
-
-
- if not isinstance( do_default_content_merge, bool ):
-
- raise HydrusExceptions.BadRequestException( 'Sorry, "do_default_content_merge" has to be a boolean! "{}" was not!'.format( do_default_content_merge ) )
-
-
- if not isinstance( delete_first, bool ):
-
- raise HydrusExceptions.BadRequestException( 'Sorry, "delete_first" has to be a boolean! "{}" was not!'.format( delete_first ) )
-
-
- if not isinstance( delete_second, bool ):
-
- raise HydrusExceptions.BadRequestException( 'Sorry, "delete_second" has to be a boolean! "{}" was not!'.format( delete_second ) )
-
-
- # ok the raw row looks good
+ hash_a = bytes.fromhex( hash_a_hex )
+ hash_b = bytes.fromhex( hash_b_hex )
list_of_service_keys_to_content_updates = []
@@ -3399,12 +3449,12 @@ class HydrusResourceClientAPIRestrictedManageFileRelationshipsSetRelationships(
list_of_service_keys_to_content_updates.append( service_keys_to_content_updates )
- rows.append( ( duplicate_type, hash_a, hash_b, list_of_service_keys_to_content_updates ) )
+ database_write_rows.append( ( duplicate_type, hash_a, hash_b, list_of_service_keys_to_content_updates ) )
- if len( rows ) > 0:
+ if len( database_write_rows ) > 0:
- HG.client_controller.WriteSynchronous( 'duplicate_pair_status', rows )
+ HG.client_controller.WriteSynchronous( 'duplicate_pair_status', database_write_rows )
response_context = HydrusServerResources.ResponseContext( 200 )
@@ -3451,38 +3501,9 @@ class HydrusResourceClientAPIRestrictedManagePagesAddFiles( HydrusResourceClient
page_key = request.parsed_request_args.GetValue( 'page_key', bytes )
- if 'hash' in request.parsed_request_args:
-
- hashes = [ request.parsed_request_args.GetValue( 'hash', bytes ) ]
-
- CheckHashLength( hashes )
-
- media_results = HG.client_controller.Read( 'media_results', hashes, sorted = True )
-
- elif 'hashes' in request.parsed_request_args:
-
- hashes = request.parsed_request_args.GetValue( 'hashes', list, expected_list_type = bytes )
-
- CheckHashLength( hashes )
-
- media_results = HG.client_controller.Read( 'media_results', hashes, sorted = True )
-
- elif 'file_id' in request.parsed_request_args:
-
- hash_ids = [ request.parsed_request_args.GetValue( 'file_id', int ) ]
-
- media_results = HG.client_controller.Read( 'media_results_from_ids', hash_ids, sorted = True )
-
- elif 'file_ids' in request.parsed_request_args:
-
- hash_ids = request.parsed_request_args.GetValue( 'file_ids', list, expected_list_type = int )
-
- media_results = HG.client_controller.Read( 'media_results_from_ids', hash_ids, sorted = True )
-
- else:
-
- raise HydrusExceptions.BadRequestException( 'You need hashes or hash_ids for this request!' )
-
+ hashes = ParseHashes( request )
+
+ media_results = HG.client_controller.Read( 'media_results', hashes, sorted = True )
try:
diff --git a/hydrus/core/HydrusConstants.py b/hydrus/core/HydrusConstants.py
index 1ed42c97..439f935a 100644
--- a/hydrus/core/HydrusConstants.py
+++ b/hydrus/core/HydrusConstants.py
@@ -84,8 +84,8 @@ options = {}
# Misc
NETWORK_VERSION = 20
-SOFTWARE_VERSION = 513
-CLIENT_API_VERSION = 40
+SOFTWARE_VERSION = 514
+CLIENT_API_VERSION = 41
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )
diff --git a/hydrus/core/HydrusData.py b/hydrus/core/HydrusData.py
index 34abf895..54449b62 100644
--- a/hydrus/core/HydrusData.py
+++ b/hydrus/core/HydrusData.py
@@ -606,6 +606,11 @@ def DebugPrint( debug_info ):
def DedupeList( xs: typing.Iterable ):
+ if isinstance( xs, set ):
+
+ return list( xs )
+
+
xs_seen = set()
xs_return = []
@@ -1381,7 +1386,7 @@ def RestartProcess():
# exe is python's exe, me is the script
- args = [ sys.executable ] + sys.argv
+ args = [ exe ] + sys.argv
else:
@@ -1398,6 +1403,7 @@ def RestartProcess():
os.execv( exe, args )
+
def SampleSetByGettingFirst( s: set, n ):
# sampling from a big set can be slow, so if we don't care about super random, let's just rip off the front and let __hash__ be our random
diff --git a/hydrus/core/HydrusText.py b/hydrus/core/HydrusText.py
index 084cfac1..78a26c4b 100644
--- a/hydrus/core/HydrusText.py
+++ b/hydrus/core/HydrusText.py
@@ -18,6 +18,7 @@ re_one_or_more_whitespace = re.compile( r'\s+' ) # this does \t and friends too
# want to keep the 'leading space' part here, despite tag.strip() elsewhere, in case of some crazy '- test' tag
re_leading_garbage = re.compile( r'^(-|system:)+' )
re_leading_single_colon = re.compile( '^:(?!:)' )
+re_leading_double_colon = re.compile( '^::(?!:)' )
re_leading_byte_order_mark = re.compile( '^\ufeff' ) # unicode .txt files prepend with this, wew
HYDRUS_NOTE_NEWLINE = '\n'
diff --git a/hydrus/core/networking/HydrusNetworkVariableHandling.py b/hydrus/core/networking/HydrusNetworkVariableHandling.py
index 52956f11..00c3d546 100644
--- a/hydrus/core/networking/HydrusNetworkVariableHandling.py
+++ b/hydrus/core/networking/HydrusNetworkVariableHandling.py
@@ -2,6 +2,7 @@ import collections
import json
import os
import traceback
+import typing
import urllib
CBOR_AVAILABLE = False
@@ -300,7 +301,7 @@ def ParseNetworkBytesToParsedHydrusArgs( network_bytes ):
return args
-def ParseTwistedRequestGETArgs( requests_args, int_params, byte_params, string_params, json_params, json_byte_list_params ):
+def ParseTwistedRequestGETArgs( requests_args: dict, int_params, byte_params, string_params, json_params, json_byte_list_params ):
args = ParsedRequestArguments()
@@ -311,9 +312,7 @@ def ParseTwistedRequestGETArgs( requests_args, int_params, byte_params, string_p
raise HydrusExceptions.NotAcceptable( 'Sorry, this service does not support CBOR!' )
- for name_bytes in requests_args:
-
- values_bytes = requests_args[ name_bytes ]
+ for ( name_bytes, values_bytes ) in requests_args.items():
try:
@@ -413,6 +412,84 @@ def ParseTwistedRequestGETArgs( requests_args, int_params, byte_params, string_p
return args
+
+variable_type_to_text_lookup = collections.defaultdict( lambda: 'unknown!' )
+
+variable_type_to_text_lookup[ int ] = 'integer'
+variable_type_to_text_lookup[ str ] = 'string'
+variable_type_to_text_lookup[ bytes ] = 'hex-encoded bytestring'
+variable_type_to_text_lookup[ bool ] = 'boolean'
+variable_type_to_text_lookup[ list ] = 'list'
+variable_type_to_text_lookup[ dict ] = 'object/dict'
+
+def GetValueFromDict( dictionary: dict, key, expected_type, expected_list_type = None, expected_dict_types = None, default_value = None, none_on_missing = False ):
+
+ # not None because in JSON sometimes people put 'null' to mean 'did not enter this optional parameter'
+ if key in dictionary and dictionary[ key ] is not None:
+
+ value = dictionary[ key ]
+
+ TestVariableType( key, value, expected_type, expected_list_type = expected_list_type, expected_dict_types = expected_dict_types )
+
+ return value
+
+ else:
+
+ if default_value is None and not none_on_missing:
+
+ raise HydrusExceptions.BadRequestException( 'The required parameter "{}" was missing!'.format( key ) )
+
+ else:
+
+ return default_value
+
+
+
+
+def TestVariableType( name: str, value: typing.Any, expected_type: type, expected_list_type = None, expected_dict_types = None, allowed_values = None ):
+
+ if not isinstance( value, expected_type ):
+
+ type_error_text = variable_type_to_text_lookup[ expected_type ]
+
+ raise HydrusExceptions.BadRequestException( 'The parameter "{}", with value "{}", was not the expected type: {}!'.format( name, value, type_error_text ) )
+
+
+ if allowed_values is not None and value not in allowed_values:
+
+ raise HydrusExceptions.BadRequestException( 'The parameter "{}", with value "{}", was not in the allowed values: {}!'.format( name, value, allowed_values ) )
+
+
+ if expected_type is list and expected_list_type is not None:
+
+ for item in value:
+
+ if not isinstance( item, expected_list_type ):
+
+ raise HydrusExceptions.BadRequestException( 'The list parameter "{}" held an item, "{}" that was {} and not the expected type: {}!'.format( name, item, type( item ), variable_type_to_text_lookup[ expected_list_type ] ) )
+
+
+
+
+ if expected_type is dict and expected_dict_types is not None:
+
+ ( expected_key_type, expected_value_type ) = expected_dict_types
+
+ for ( dict_key, dict_value ) in value.items():
+
+ if not isinstance( dict_key, expected_key_type ):
+
+ raise HydrusExceptions.BadRequestException( 'The Object parameter "{}" held a key, "{}" that was {} and not the expected type: {}!'.format( name, dict_key, type( dict_key ), variable_type_to_text_lookup[ expected_key_type ] ) )
+
+
+ if not isinstance( dict_value, expected_value_type ):
+
+ raise HydrusExceptions.BadRequestException( 'The Object parameter "{}" held a value, "{}" that was {} and not the expected type: {}!'.format( name, dict_value, type( dict_value ), variable_type_to_text_lookup[ expected_value_type ] ) )
+
+
+
+
+
class ParsedRequestArguments( dict ):
def __missing__( self, key ):
@@ -422,75 +499,6 @@ class ParsedRequestArguments( dict ):
def GetValue( self, key, expected_type, expected_list_type = None, expected_dict_types = None, default_value = None, none_on_missing = False ):
- # not None because in JSON sometimes people put 'null' to mean 'did not enter this optional parameter'
- if key in self and self[ key ] is not None:
-
- value = self[ key ]
-
- error_text_lookup = collections.defaultdict( lambda: 'unknown!' )
-
- error_text_lookup[ int ] = 'integer'
- error_text_lookup[ str ] = 'string'
- error_text_lookup[ bytes ] = 'hex-encoded bytestring'
- error_text_lookup[ bool ] = 'boolean'
- error_text_lookup[ list ] = 'list'
- error_text_lookup[ dict ] = 'object/dict'
-
- if not isinstance( value, expected_type ):
-
- if expected_type in error_text_lookup:
-
- type_error_text = error_text_lookup[ expected_type ]
-
- else:
-
- type_error_text = 'unknown!'
-
-
- raise HydrusExceptions.BadRequestException( 'The parameter "{}" was not the expected type: {}!'.format( key, type_error_text ) )
-
-
- if expected_type is list and expected_list_type is not None:
-
- for item in value:
-
- if not isinstance( item, expected_list_type ):
-
- raise HydrusExceptions.BadRequestException( 'The list parameter "{}" held an item, "{}" that was {} and not the expected type: {}!'.format( key, item, type( item ), error_text_lookup[ expected_list_type ] ) )
-
-
-
-
- if expected_type is dict and expected_dict_types is not None:
-
- ( expected_key_type, expected_value_type ) = expected_dict_types
-
- for ( dict_key, dict_value ) in value.items():
-
- if not isinstance( dict_key, expected_key_type ):
-
- raise HydrusExceptions.BadRequestException( 'The Object parameter "{}" held a key, "{}" that was {} and not the expected type: {}!'.format( key, dict_key, type( dict_key ), error_text_lookup[ expected_key_type ] ) )
-
-
- if not isinstance( dict_value, expected_value_type ):
-
- raise HydrusExceptions.BadRequestException( 'The Object parameter "{}" held a value, "{}" that was {} and not the expected type: {}!'.format( key, dict_value, type( dict_value ), error_text_lookup[ expected_value_type ] ) )
-
-
-
-
- return value
-
- else:
-
- if default_value is None and not none_on_missing:
-
- raise HydrusExceptions.BadRequestException( 'The required parameter "{}" was missing!'.format( key ) )
-
- else:
-
- return default_value
-
-
+ return GetValueFromDict( self, key, expected_type, expected_list_type = expected_list_type, expected_dict_types = expected_dict_types, default_value = default_value, none_on_missing = none_on_missing )
diff --git a/hydrus/test/TestClientAPI.py b/hydrus/test/TestClientAPI.py
index 118170a0..017c718c 100644
--- a/hydrus/test/TestClientAPI.py
+++ b/hydrus/test/TestClientAPI.py
@@ -452,7 +452,7 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_tags/add_tags'
- body_dict = { 'Hydrus-Client-API-Access-Key' : 'abcd', 'hash' : hash_hex, 'service_names_to_tags' : { 'my tags' : [ 'test', 'test2' ] } }
+ body_dict = { 'Hydrus-Client-API-Access-Key' : 'abcd', 'hash' : hash_hex, 'service_keys_to_tags' : { CC.DEFAULT_LOCAL_TAG_SERVICE_KEY.hex() : [ 'test', 'test2' ] } }
body = json.dumps( body_dict )
@@ -464,7 +464,7 @@ class TestClientAPI( unittest.TestCase ):
self.assertEqual( response.status, 403 )
- body_dict = { 'Hydrus-Client-API-Session-Key' : 'abcd', 'hash' : hash_hex, 'service_names_to_tags' : { 'my tags' : [ 'test', 'test2' ] } }
+ body_dict = { 'Hydrus-Client-API-Session-Key' : 'abcd', 'hash' : hash_hex, 'service_keys_to_tags' : { CC.DEFAULT_LOCAL_TAG_SERVICE_KEY.hex() : [ 'test', 'test2' ] } }
body = json.dumps( body_dict )
@@ -505,7 +505,7 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_tags/add_tags'
- body_dict = { 'Hydrus-Client-API-Access-Key' : access_key_hex, 'hash' : hash_hex, 'service_names_to_tags' : { 'my tags' : [ 'test', 'test2' ] } }
+ body_dict = { 'Hydrus-Client-API-Access-Key' : access_key_hex, 'hash' : hash_hex, 'service_keys_to_tags' : { CC.DEFAULT_LOCAL_TAG_SERVICE_KEY.hex() : [ 'test', 'test2' ] } }
body = json.dumps( body_dict )
@@ -521,7 +521,7 @@ class TestClientAPI( unittest.TestCase ):
HG.test_controller.ClearWrites( 'content_updates' )
- body_dict = { 'Hydrus-Client-API-Session-Key' : session_key_hex, 'hash' : hash_hex, 'service_names_to_tags' : { 'my tags' : [ 'test', 'test2' ] } }
+ body_dict = { 'Hydrus-Client-API-Session-Key' : session_key_hex, 'hash' : hash_hex, 'service_keys_to_tags' : { CC.DEFAULT_LOCAL_TAG_SERVICE_KEY.hex() : [ 'test', 'test2' ] } }
body = json.dumps( body_dict )
@@ -728,6 +728,15 @@ class TestClientAPI( unittest.TestCase ):
]
}
+ get_service_expected_result = {
+ 'service' : {
+ 'name' : 'repository updates',
+ 'service_key' : '7265706f7369746f72792075706461746573',
+ 'type': 20,
+ 'type_pretty': 'local update file domain'
+ }
+ }
+
for api_permissions in should_work.union( should_break ):
access_key_hex = api_permissions.GetAccessKey().hex()
@@ -759,6 +768,55 @@ class TestClientAPI( unittest.TestCase ):
self.assertEqual( response.status, 403 )
+ #
+
+ path = '/get_service?service_name=repository%20updates'
+
+ connection.request( 'GET', path, headers = headers )
+
+ response = connection.getresponse()
+
+ data = response.read()
+
+ if api_permissions in should_work:
+
+ text = str( data, 'utf-8' )
+
+ self.assertEqual( response.status, 200 )
+
+ d = json.loads( text )
+
+ self.assertEqual( d, get_service_expected_result )
+
+ else:
+
+ self.assertEqual( response.status, 403 )
+
+
+ path = '/get_service?service_key={}'.format( CC.LOCAL_UPDATE_SERVICE_KEY.hex() )
+
+ connection.request( 'GET', path, headers = headers )
+
+ response = connection.getresponse()
+
+ data = response.read()
+
+ if api_permissions in should_work:
+
+ text = str( data, 'utf-8' )
+
+ self.assertEqual( response.status, 200 )
+
+ d = json.loads( text )
+
+ self.assertEqual( d, get_service_expected_result )
+
+ else:
+
+ self.assertEqual( response.status, 403 )
+
+
+
def _test_add_files_add_file( self, connection, set_up_permissions ):
@@ -966,7 +1024,9 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_files/delete_files'
- body_dict = { 'hashes' : [ h.hex() for h in hashes ], 'file_service_name' : 'not existing service' }
+ not_existing_service_hex = os.urandom( 32 ).hex()
+
+ body_dict = { 'hashes' : [ h.hex() for h in hashes ], 'file_service_key' : not_existing_service_hex }
body = json.dumps( body_dict )
@@ -980,7 +1040,7 @@ class TestClientAPI( unittest.TestCase ):
text = str( data, 'utf-8' )
- self.assertIn( 'not existing service', text ) # error message should be complaining about it
+ self.assertIn( not_existing_service_hex, text ) # error message should be complaining about it
#
@@ -1036,7 +1096,7 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_files/undelete_files'
- body_dict = { 'hashes' : [ h.hex() for h in hashes ], 'file_service_name' : 'not existing service' }
+ body_dict = { 'hashes' : [ h.hex() for h in hashes ], 'file_service_key' : not_existing_service_hex }
body = json.dumps( body_dict )
@@ -1050,7 +1110,7 @@ class TestClientAPI( unittest.TestCase ):
text = str( data, 'utf-8' )
- self.assertIn( 'not existing service', text ) # error message should be complaining about it
+ self.assertIn( not_existing_service_hex, text ) # error message should be complaining about it
#
@@ -1482,7 +1542,7 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_tags/add_tags'
- body_dict = { 'service_names_to_tags' : { 'my tags' : [ 'test' ] } }
+ body_dict = { 'service_keys_to_tags' : { CC.DEFAULT_LOCAL_TAG_SERVICE_KEY.hex() : [ 'test' ] } }
body = json.dumps( body_dict )
@@ -1498,7 +1558,9 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_tags/add_tags'
- body_dict = { 'hash' : hash_hex, 'service_names_to_tags' : { 'bad tag service' : [ 'test' ] } }
+ not_existing_service_key_hex = os.urandom( 32 ).hex()
+
+ body_dict = { 'hash' : hash_hex, 'service_keys_to_tags' : { not_existing_service_key_hex : [ 'test' ] } }
body = json.dumps( body_dict )
@@ -1510,13 +1572,17 @@ class TestClientAPI( unittest.TestCase ):
self.assertEqual( response.status, 400 )
+ text = str( data, 'utf-8' )
+
+ self.assertIn( not_existing_service_key_hex, text ) # test it complains about the key in the error
+
# add tags to local
HG.test_controller.ClearWrites( 'content_updates' )
path = '/add_tags/add_tags'
- body_dict = { 'hash' : hash_hex, 'service_names_to_tags' : { 'my tags' : [ 'test', 'test2' ] } }
+ body_dict = { 'hash' : hash_hex, 'service_keys_to_tags' : { CC.DEFAULT_LOCAL_TAG_SERVICE_KEY.hex() : [ 'test', 'test2' ] } }
body = json.dumps( body_dict )
@@ -1542,7 +1608,7 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_tags/add_tags'
- body_dict = { 'hash' : hash_hex, 'service_names_to_actions_to_tags' : { 'my tags' : { str( HC.CONTENT_UPDATE_ADD ) : [ 'test_add', 'test_add2' ], str( HC.CONTENT_UPDATE_DELETE ) : [ 'test_delete', 'test_delete2' ] } } }
+ body_dict = { 'hash' : hash_hex, 'service_keys_to_actions_to_tags' : { CC.DEFAULT_LOCAL_TAG_SERVICE_KEY.hex() : { str( HC.CONTENT_UPDATE_ADD ) : [ 'test_add', 'test_add2' ], str( HC.CONTENT_UPDATE_DELETE ) : [ 'test_delete', 'test_delete2' ] } } }
body = json.dumps( body_dict )
@@ -1573,7 +1639,7 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_tags/add_tags'
- body_dict = { 'hash' : hash_hex, 'service_names_to_tags' : { 'example tag repo' : [ 'test', 'test2' ] } }
+ body_dict = { 'hash' : hash_hex, 'service_keys_to_tags' : { HG.test_controller.example_tag_repo_service_key.hex() : [ 'test', 'test2' ] } }
body = json.dumps( body_dict )
@@ -1599,7 +1665,7 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_tags/add_tags'
- body_dict = { 'hash' : hash_hex, 'service_names_to_actions_to_tags' : { 'example tag repo' : { str( HC.CONTENT_UPDATE_PEND ) : [ 'test_add', 'test_add2' ], str( HC.CONTENT_UPDATE_PETITION ) : [ [ 'test_delete', 'muh reason' ], 'test_delete2' ] } } }
+ body_dict = { 'hash' : hash_hex, 'service_keys_to_actions_to_tags' : { HG.test_controller.example_tag_repo_service_key.hex() : { str( HC.CONTENT_UPDATE_PEND ) : [ 'test_add', 'test_add2' ], str( HC.CONTENT_UPDATE_PETITION ) : [ [ 'test_delete', 'muh reason' ], 'test_delete2' ] } } }
body = json.dumps( body_dict )
@@ -1630,7 +1696,7 @@ class TestClientAPI( unittest.TestCase ):
path = '/add_tags/add_tags'
- body_dict = { 'hashes' : [ hash_hex, hash2_hex ], 'service_names_to_tags' : { 'my tags' : [ 'test', 'test2' ] } }
+ body_dict = { 'hashes' : [ hash_hex, hash2_hex ], 'service_keys_to_tags' : { CC.DEFAULT_LOCAL_TAG_SERVICE_KEY.hex() : [ 'test', 'test2' ] } }
body = json.dumps( body_dict )
@@ -2080,7 +2146,7 @@ class TestClientAPI( unittest.TestCase ):
HG.test_controller.ClearWrites( 'import_url_test' )
- request_dict = { 'url' : url, 'destination_page_name' : 'muh /tv/', 'show_destination_page' : True, 'filterable_tags' : [ 'filename:yo' ], 'service_names_to_additional_tags' : { 'my tags' : [ '/tv/ thread' ] } }
+ request_dict = { 'url' : url, 'destination_page_name' : 'muh /tv/', 'show_destination_page' : True, 'filterable_tags' : [ 'filename:yo' ], 'service_keys_to_additional_tags' : { CC.DEFAULT_LOCAL_TAG_SERVICE_KEY.hex() : [ '/tv/ thread' ] } }
request_body = json.dumps( request_dict )
@@ -2570,7 +2636,7 @@ class TestClientAPI( unittest.TestCase ):
( location_context, hashes ) = args
self.assertEqual( location_context, default_location_context )
- self.assertEqual( hashes, { file_relationships_hash } )
+ self.assertEqual( set( hashes ), { file_relationships_hash } )
# search files failed tag permission
@@ -2850,13 +2916,30 @@ class TestClientAPI( unittest.TestCase ):
path = '/manage_file_relationships/set_file_relationships'
- test_pair_rows = [
- [ 4, "b54d09218e0d6efc964b78b070620a1fa19c7e069672b4c6313cee2c9b0623f2", "bbaa9876dab238dcf5799bfd8319ed0bab805e844f45cf0de33f40697b11a845", False, False, True ],
- [ 4, "22667427eaa221e2bd7ef405e1d2983846c863d40b2999ce8d1bf5f0c18f5fb2", "65d228adfa722f3cd0363853a191898abe8bf92d9a514c6c7f3c89cfed0bf423", False, False, True ],
- [ 2, "0480513ffec391b77ad8c4e57fe80e5b710adfa3cb6af19b02a0bd7920f2d3ec", "5fab162576617b5c3fc8caabea53ce3ab1a3c8e0a16c16ae7b4e4a21eab168a7", False, False, False ]
- ]
-
- request_dict = { 'pair_rows' : test_pair_rows }
+ request_dict = {
+ "relationships" : [
+ {
+ "hash_a" : "b54d09218e0d6efc964b78b070620a1fa19c7e069672b4c6313cee2c9b0623f2",
+ "hash_b" : "bbaa9876dab238dcf5799bfd8319ed0bab805e844f45cf0de33f40697b11a845",
+ "relationship" : 4,
+ "do_default_content_merge" : False,
+ "delete_b" : True
+ },
+ {
+ "hash_a" : "22667427eaa221e2bd7ef405e1d2983846c863d40b2999ce8d1bf5f0c18f5fb2",
+ "hash_b" : "65d228adfa722f3cd0363853a191898abe8bf92d9a514c6c7f3c89cfed0bf423",
+ "relationship" : 4,
+ "do_default_content_merge" : False,
+ "delete_b" : True
+ },
+ {
+ "hash_a" : "0480513ffec391b77ad8c4e57fe80e5b710adfa3cb6af19b02a0bd7920f2d3ec",
+ "hash_b" : "5fab162576617b5c3fc8caabea53ce3ab1a3c8e0a16c16ae7b4e4a21eab168a7",
+ "relationship" : 2,
+ "do_default_content_merge" : False
+ }
+ ]
+ }
request_body = json.dumps( request_dict )
@@ -2888,7 +2971,7 @@ class TestClientAPI( unittest.TestCase ):
- expected_written_rows = [ ( duplicate_type, bytes.fromhex( hash_a_hex ), bytes.fromhex( hash_b_hex ), delete_thing( hash_b_hex, delete_second ) ) for ( duplicate_type, hash_a_hex, hash_b_hex, merge, delete_first, delete_second ) in test_pair_rows ]
+ expected_written_rows = [ ( r_dict[ 'relationship' ], bytes.fromhex( r_dict[ 'hash_a' ] ), bytes.fromhex( r_dict[ 'hash_b' ] ), delete_thing( r_dict[ 'hash_b' ], 'delete_b' in r_dict and r_dict[ 'delete_b' ] ) ) for r_dict in request_dict[ 'relationships' ] ]
self.assertEqual( written_rows, expected_written_rows )
@@ -3294,11 +3377,11 @@ class TestClientAPI( unittest.TestCase ):
tags = [ 'kino', 'green' ]
- path = '/get_files/search_files?tags={}&file_sort_type={}&file_sort_asc={}&file_service_name={}'.format(
+ path = '/get_files/search_files?tags={}&file_sort_type={}&file_sort_asc={}&file_service_key={}'.format(
urllib.parse.quote( json.dumps( tags ) ),
CC.SORT_FILES_BY_FRAMERATE,
'true',
- 'trash'
+ CC.TRASH_SERVICE_KEY.hex()
)
connection.request( 'GET', path, headers = headers )
@@ -3340,12 +3423,12 @@ class TestClientAPI( unittest.TestCase ):
tags = [ 'kino', 'green' ]
- path = '/get_files/search_files?tags={}&file_sort_type={}&file_sort_asc={}&file_service_key={}&tag_service_name={}'.format(
+ path = '/get_files/search_files?tags={}&file_sort_type={}&file_sort_asc={}&file_service_key={}&tag_service_key={}'.format(
urllib.parse.quote( json.dumps( tags ) ),
CC.SORT_FILES_BY_FRAMERATE,
'true',
CC.TRASH_SERVICE_KEY.hex(),
- 'all%20known%20tags'
+ CC.COMBINED_TAG_SERVICE_KEY.hex()
)
connection.request( 'GET', path, headers = headers )
@@ -3449,18 +3532,6 @@ class TestClientAPI( unittest.TestCase ):
self.assertEqual( predicates, [] )
- #
-
- pretend_request = PretendRequest()
-
- pretend_request.parsed_request_args = { 'system_inbox' : True }
- pretend_request.client_api_permissions = set_up_permissions[ 'search_green_files' ]
-
- with self.assertRaises( HydrusExceptions.InsufficientCredentialsException ):
-
- ClientLocalServerResources.ParseClientAPISearchPredicates( pretend_request )
-
-
#
pretend_request = PretendRequest()
@@ -3517,38 +3588,6 @@ class TestClientAPI( unittest.TestCase ):
pretend_request = PretendRequest()
- pretend_request.parsed_request_args = { 'tags' : [ 'green' ], 'system_inbox' : True }
- pretend_request.client_api_permissions = set_up_permissions[ 'search_green_files' ]
-
- predicates = ClientLocalServerResources.ParseClientAPISearchPredicates( pretend_request )
-
- expected_predicates = []
-
- expected_predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_TAG, value = 'green' ) )
- expected_predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_SYSTEM_INBOX ) )
-
- self.assertEqual( set( predicates ), set( expected_predicates ) )
-
- #
-
- pretend_request = PretendRequest()
-
- pretend_request.parsed_request_args = { 'tags' : [ 'green' ], 'system_archive' : True }
- pretend_request.client_api_permissions = set_up_permissions[ 'search_green_files' ]
-
- predicates = ClientLocalServerResources.ParseClientAPISearchPredicates( pretend_request )
-
- expected_predicates = []
-
- expected_predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_TAG, value = 'green' ) )
- expected_predicates.append( ClientSearch.Predicate( predicate_type = ClientSearch.PREDICATE_TYPE_SYSTEM_ARCHIVE ) )
-
- self.assertEqual( set( predicates ), set( expected_predicates ) )
-
- #
-
- pretend_request = PretendRequest()
-
pretend_request.parsed_request_args = { 'tags' : [ 'green', 'system:archive' ] }
pretend_request.client_api_permissions = set_up_permissions[ 'search_green_files' ]
@@ -3676,12 +3715,17 @@ class TestClientAPI( unittest.TestCase ):
headers = { 'Hydrus-Client-API-Access-Key' : access_key_hex }
- file_ids_to_hashes = { 1 : bytes.fromhex( 'a' * 64 ), 2 : bytes.fromhex( 'b' * 64 ), 3 : bytes.fromhex( 'c' * 64 ) }
+ file_ids_to_hashes = { i : os.urandom( 32 ) for i in range( 20 ) }
metadata = []
for ( file_id, hash ) in file_ids_to_hashes.items():
+ if file_id == 0 or file_id >= 4:
+
+ continue
+
+
metadata_row = { 'file_id' : file_id, 'hash' : hash.hex() }
metadata.append( metadata_row )
@@ -3709,6 +3753,11 @@ class TestClientAPI( unittest.TestCase ):
for ( file_id, hash ) in file_ids_to_hashes.items():
+ if file_id == 0 or file_id >= 4:
+
+ continue
+
+
size = random.randint( 8192, 20 * 1048576 )
mime = random.choice( [ HC.IMAGE_JPEG, HC.VIDEO_WEBM, HC.APPLICATION_PDF ] )
width = random.randint( 200, 4096 )
@@ -3888,56 +3937,6 @@ class TestClientAPI( unittest.TestCase ):
metadata_row[ 'tags' ] = tags_dict
- # old stuff start
-
- api_service_keys_to_statuses_to_tags = {}
-
- service_keys_to_statuses_to_tags = tags_manager.GetServiceKeysToStatusesToTags( ClientTags.TAG_DISPLAY_STORAGE )
-
- for ( service_key, statuses_to_tags ) in service_keys_to_statuses_to_tags.items():
-
- if service_key not in service_keys_to_names:
-
- service_keys_to_names[ service_key ] = services_manager.GetName( service_key )
-
-
- s = { str( status ) : sorted( tags, key = HydrusTags.ConvertTagToSortable ) for ( status, tags ) in statuses_to_tags.items() if len( tags ) > 0 }
-
- if len( s ) > 0:
-
- service_name = service_keys_to_names[ service_key ]
-
- api_service_keys_to_statuses_to_tags[ service_key.hex() ] = s
-
-
-
- metadata_row[ 'service_keys_to_statuses_to_tags' ] = api_service_keys_to_statuses_to_tags
-
- service_keys_to_statuses_to_display_tags = {}
-
- service_keys_to_statuses_to_tags = tags_manager.GetServiceKeysToStatusesToTags( ClientTags.TAG_DISPLAY_ACTUAL )
-
- for ( service_key, statuses_to_tags ) in service_keys_to_statuses_to_tags.items():
-
- if service_key not in service_keys_to_names:
-
- service_keys_to_names[ service_key ] = services_manager.GetName( service_key )
-
-
- s = { str( status ) : sorted( tags, key = HydrusTags.ConvertTagToSortable ) for ( status, tags ) in statuses_to_tags.items() if len( tags ) > 0 }
-
- if len( s ) > 0:
-
- service_name = service_keys_to_names[ service_key ]
-
- service_keys_to_statuses_to_display_tags[ service_key.hex() ] = s
-
-
-
- metadata_row[ 'service_keys_to_statuses_to_display_tags' ] = service_keys_to_statuses_to_display_tags
-
- # old stuff end
-
metadata.append( metadata_row )
detailed_known_urls_metadata_row = dict( metadata_row )
@@ -3971,6 +3970,8 @@ class TestClientAPI( unittest.TestCase ):
# fail on non-permitted files
+ HG.test_controller.SetRead( 'hash_ids_to_hashes', { k : v for ( k, v ) in file_ids_to_hashes.items() if k in [ 1, 2, 3, 7 ] } )
+
path = '/get_files/file_metadata?file_ids={}&only_return_identifiers=true'.format( urllib.parse.quote( json.dumps( [ 1, 2, 3, 7 ] ) ) )
connection.request( 'GET', path, headers = headers )
@@ -3995,6 +3996,8 @@ class TestClientAPI( unittest.TestCase ):
# identifiers from file_ids
+ HG.test_controller.SetRead( 'hash_ids_to_hashes', { k : v for ( k, v ) in file_ids_to_hashes.items() if k in [ 1, 2, 3 ] } )
+
path = '/get_files/file_metadata?file_ids={}&only_return_identifiers=true'.format( urllib.parse.quote( json.dumps( [ 1, 2, 3 ] ) ) )
connection.request( 'GET', path, headers = headers )
@@ -4013,6 +4016,8 @@ class TestClientAPI( unittest.TestCase ):
# basic metadata from file_ids
+ HG.test_controller.SetRead( 'hash_ids_to_hashes', { k : v for ( k, v ) in file_ids_to_hashes.items() if k in [ 1, 2, 3 ] } )
+
path = '/get_files/file_metadata?file_ids={}&only_return_basic_information=true'.format( urllib.parse.quote( json.dumps( [ 1, 2, 3 ] ) ) )
connection.request( 'GET', path, headers = headers )
@@ -4031,6 +4036,8 @@ class TestClientAPI( unittest.TestCase ):
# metadata from file_ids
+ HG.test_controller.SetRead( 'hash_ids_to_hashes', { k : v for ( k, v ) in file_ids_to_hashes.items() if k in [ 1, 2, 3 ] } )
+
path = '/get_files/file_metadata?file_ids={}'.format( urllib.parse.quote( json.dumps( [ 1, 2, 3 ] ) ) )
connection.request( 'GET', path, headers = headers )
@@ -4084,7 +4091,7 @@ class TestClientAPI( unittest.TestCase ):
# identifiers from hashes
- path = '/get_files/file_metadata?hashes={}&only_return_identifiers=true'.format( urllib.parse.quote( json.dumps( [ hash.hex() for hash in file_ids_to_hashes.values() ] ) ) )
+ path = '/get_files/file_metadata?hashes={}&only_return_identifiers=true'.format( urllib.parse.quote( json.dumps( [ hash.hex() for ( k, hash ) in file_ids_to_hashes.items() if k in [ 1, 2, 3 ] ] ) ) )
connection.request( 'GET', path, headers = headers )
@@ -4102,7 +4109,7 @@ class TestClientAPI( unittest.TestCase ):
# basic metadata from hashes
- path = '/get_files/file_metadata?hashes={}&only_return_basic_information=true'.format( urllib.parse.quote( json.dumps( [ hash.hex() for hash in file_ids_to_hashes.values() ] ) ) )
+ path = '/get_files/file_metadata?hashes={}&only_return_basic_information=true'.format( urllib.parse.quote( json.dumps( [ hash.hex() for ( k, hash ) in file_ids_to_hashes.items() if k in [ 1, 2, 3 ] ] ) ) )
connection.request( 'GET', path, headers = headers )
@@ -4120,7 +4127,7 @@ class TestClientAPI( unittest.TestCase ):
# metadata from hashes
- path = '/get_files/file_metadata?hashes={}'.format( urllib.parse.quote( json.dumps( [ hash.hex() for hash in file_ids_to_hashes.values() ] ) ) )
+ path = '/get_files/file_metadata?hashes={}'.format( urllib.parse.quote( json.dumps( [ hash.hex() for ( k, hash ) in file_ids_to_hashes.items() if k in [ 1, 2, 3 ] ] ) ) )
connection.request( 'GET', path, headers = headers )
@@ -4150,7 +4157,7 @@ class TestClientAPI( unittest.TestCase ):
# metadata from hashes with detailed url info
- path = '/get_files/file_metadata?hashes={}&detailed_url_information=true'.format( urllib.parse.quote( json.dumps( [ hash.hex() for hash in file_ids_to_hashes.values() ] ) ) )
+ path = '/get_files/file_metadata?hashes={}&detailed_url_information=true'.format( urllib.parse.quote( json.dumps( [ hash.hex() for ( k, hash ) in file_ids_to_hashes.items() if k in [ 1, 2, 3 ] ] ) ) )
connection.request( 'GET', path, headers = headers )
@@ -4168,7 +4175,7 @@ class TestClientAPI( unittest.TestCase ):
# metadata from hashes with notes info
- path = '/get_files/file_metadata?hashes={}&include_notes=true'.format( urllib.parse.quote( json.dumps( [ hash.hex() for hash in file_ids_to_hashes.values() ] ) ) )
+ path = '/get_files/file_metadata?hashes={}&include_notes=true'.format( urllib.parse.quote( json.dumps( [ hash.hex() for ( k, hash ) in file_ids_to_hashes.items() if k in [ 1, 2, 3 ] ] ) ) )
connection.request( 'GET', path, headers = headers )
@@ -4186,7 +4193,7 @@ class TestClientAPI( unittest.TestCase ):
# failure on missing file_ids
- HG.test_controller.SetRead( 'media_results_from_ids', HydrusExceptions.DataMissing( 'test missing' ) )
+ HG.test_controller.SetRead( 'hash_ids_to_hashes', HydrusExceptions.DataMissing( 'test missing' ) )
api_permissions = set_up_permissions[ 'everything' ]
diff --git a/hydrus/test/TestClientDBDuplicates.py b/hydrus/test/TestClientDBDuplicates.py
index 4c245974..531b16d2 100644
--- a/hydrus/test/TestClientDBDuplicates.py
+++ b/hydrus/test/TestClientDBDuplicates.py
@@ -749,7 +749,9 @@ class TestClientDBDuplicates( unittest.TestCase ):
self.assertEqual( len( file_duplicate_types_to_counts ), 5 )
- self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], self._get_group_potential_count( file_duplicate_types_to_counts ) )
+ expected = self._get_group_potential_count( file_duplicate_types_to_counts )
+
+ self.assertIn( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], ( expected, expected - 1 ) )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_MEMBER ], len( self._our_main_dupe_group_hashes ) - 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_FALSE_POSITIVE ], 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_ALTERNATE ], 1 )
@@ -765,7 +767,9 @@ class TestClientDBDuplicates( unittest.TestCase ):
self.assertEqual( len( file_duplicate_types_to_counts ), 5 )
- self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], self._get_group_potential_count( file_duplicate_types_to_counts ) )
+ expected = self._get_group_potential_count( file_duplicate_types_to_counts )
+
+ self.assertIn( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], ( expected, expected - 1 ) )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_MEMBER ], len( self._our_alt_dupe_group_hashes ) - 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_FALSE_POSITIVE ], 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_ALTERNATE ], 1 )
@@ -810,7 +814,9 @@ class TestClientDBDuplicates( unittest.TestCase ):
self.assertEqual( len( file_duplicate_types_to_counts ), 5 )
- self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], self._get_group_potential_count( file_duplicate_types_to_counts ) )
+ expected = self._get_group_potential_count( file_duplicate_types_to_counts )
+
+ self.assertIn( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], ( expected, expected - 1 ) )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_MEMBER ], len( self._our_main_dupe_group_hashes ) - 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_FALSE_POSITIVE ], 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_ALTERNATE ], 1 )
@@ -824,7 +830,9 @@ class TestClientDBDuplicates( unittest.TestCase ):
self.assertEqual( len( file_duplicate_types_to_counts ), 3 )
- self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], self._get_group_potential_count( file_duplicate_types_to_counts ) )
+ expected = self._get_group_potential_count( file_duplicate_types_to_counts )
+
+ self.assertIn( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], ( expected, expected - 1 ) )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_MEMBER ], len( self._our_fp_dupe_group_hashes ) - 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_FALSE_POSITIVE ], 2 )
@@ -869,7 +877,9 @@ class TestClientDBDuplicates( unittest.TestCase ):
self.assertEqual( len( file_duplicate_types_to_counts ), 5 )
- self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], self._get_group_potential_count( file_duplicate_types_to_counts ) )
+ expected = self._get_group_potential_count( file_duplicate_types_to_counts )
+
+ self.assertIn( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], ( expected, expected - 1 ) )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_MEMBER ], len( self._our_main_dupe_group_hashes ) - 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_FALSE_POSITIVE ], 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_ALTERNATE ], 1 )
@@ -883,7 +893,9 @@ class TestClientDBDuplicates( unittest.TestCase ):
self.assertEqual( len( file_duplicate_types_to_counts ), 5 )
- self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], self._get_group_potential_count( file_duplicate_types_to_counts ) )
+ expected = self._get_group_potential_count( file_duplicate_types_to_counts )
+
+ self.assertIn( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], ( expected, expected - 1 ) )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_MEMBER ], len( self._our_alt_dupe_group_hashes ) - 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_FALSE_POSITIVE ], 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_ALTERNATE ], 1 )
@@ -906,7 +918,9 @@ class TestClientDBDuplicates( unittest.TestCase ):
self.assertEqual( len( file_duplicate_types_to_counts ), 5 )
- self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], self._get_group_potential_count( file_duplicate_types_to_counts ) )
+ expected = self._get_group_potential_count( file_duplicate_types_to_counts )
+
+ self.assertIn( file_duplicate_types_to_counts[ HC.DUPLICATE_POTENTIAL ], ( expected, expected - 1 ) )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_MEMBER ], len( self._our_main_dupe_group_hashes ) - 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_FALSE_POSITIVE ], 1 )
self.assertEqual( file_duplicate_types_to_counts[ HC.DUPLICATE_ALTERNATE ], 1 )
diff --git a/hydrus/test/TestClientTags.py b/hydrus/test/TestClientTags.py
index 0727bb12..130776f0 100644
--- a/hydrus/test/TestClientTags.py
+++ b/hydrus/test/TestClientTags.py
@@ -461,9 +461,9 @@ class TestTagObjects( unittest.TestCase ):
self.assertEqual( pat.IsAcceptableForFileSearches(), values[0] )
self.assertEqual( pat.IsAcceptableForTagSearches(), values[1] )
self.assertEqual( pat.IsEmpty(), values[2] )
- self.assertEqual( pat.IsExplicitWildcard(), values[3] )
+ self.assertEqual( pat.IsExplicitWildcard( True ), values[3] )
self.assertEqual( pat.IsNamespaceSearch(), values[4] )
- self.assertEqual( pat.IsTagSearch(), values[5] )
+ self.assertEqual( pat.IsTagSearch( True ), values[5] )
self.assertEqual( pat.inclusive, values[6] )
@@ -475,8 +475,8 @@ class TestTagObjects( unittest.TestCase ):
def read_predicate_tests( pat: ClientSearch.ParsedAutocompleteText, values ):
- self.assertEqual( pat.GetImmediateFileSearchPredicate(), values[0] )
- self.assertEqual( pat.GetNonTagFileSearchPredicates(), values[1] )
+ self.assertEqual( pat.GetImmediateFileSearchPredicate( True ), values[0] )
+ self.assertEqual( pat.GetNonTagFileSearchPredicates( True ), values[1] )
def write_predicate_tests( pat: ClientSearch.ParsedAutocompleteText, values ):
diff --git a/mkdocs.yml b/mkdocs.yml
index 317de3fe..c8d96f14 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -27,6 +27,7 @@ nav:
- Advanced:
- advanced_siblings.md
- advanced_parents.md
+ - advanced_sidecars.md
- advanced_multiple_local_file_services.md
- advanced.md
- reducing_lag.md
diff --git a/static/default/gugs/twitter collection lookup.png b/static/default/gugs/twitter collection lookup.png
deleted file mode 100644
index c8b0328c..00000000
Binary files a/static/default/gugs/twitter collection lookup.png and /dev/null differ
diff --git a/static/default/gugs/twitter likes lookup.png b/static/default/gugs/twitter likes lookup.png
deleted file mode 100644
index b4c68da5..00000000
Binary files a/static/default/gugs/twitter likes lookup.png and /dev/null differ
diff --git a/static/default/gugs/twitter list lookup.png b/static/default/gugs/twitter list lookup.png
deleted file mode 100644
index 6377215a..00000000
Binary files a/static/default/gugs/twitter list lookup.png and /dev/null differ
diff --git a/static/default/gugs/twitter profile lookup (with replies).png b/static/default/gugs/twitter profile lookup (with replies).png
index 8a25eb2f..663b943b 100644
Binary files a/static/default/gugs/twitter profile lookup (with replies).png and b/static/default/gugs/twitter profile lookup (with replies).png differ
diff --git a/static/default/gugs/twitter profile lookup.png b/static/default/gugs/twitter profile lookup.png
index 36bdf549..6b5cf21b 100644
Binary files a/static/default/gugs/twitter profile lookup.png and b/static/default/gugs/twitter profile lookup.png differ
diff --git a/static/default/parsers/danbooru file page parser - get webm ugoira.png b/static/default/parsers/danbooru file page parser - get webm ugoira.png
index ecabe788..bac3157c 100644
Binary files a/static/default/parsers/danbooru file page parser - get webm ugoira.png and b/static/default/parsers/danbooru file page parser - get webm ugoira.png differ
diff --git a/static/default/parsers/deviant art file page parser.png b/static/default/parsers/deviant art file page parser.png
index acc35245..d38ff68a 100644
Binary files a/static/default/parsers/deviant art file page parser.png and b/static/default/parsers/deviant art file page parser.png differ
diff --git a/static/default/parsers/pixiv file page api parser.png b/static/default/parsers/pixiv file page api parser.png
index 5940568f..f5e85086 100644
Binary files a/static/default/parsers/pixiv file page api parser.png and b/static/default/parsers/pixiv file page api parser.png differ
diff --git a/static/default/parsers/twitter syndication api timeline-profile parser.png b/static/default/parsers/twitter syndication api timeline-profile parser.png
new file mode 100644
index 00000000..d53f955c
Binary files /dev/null and b/static/default/parsers/twitter syndication api timeline-profile parser.png differ
diff --git a/static/default/parsers/twitter syndication api tweet parser.png b/static/default/parsers/twitter syndication api tweet parser.png
index dac8ea7b..7db74aad 100644
Binary files a/static/default/parsers/twitter syndication api tweet parser.png and b/static/default/parsers/twitter syndication api tweet parser.png differ
diff --git a/static/default/url_classes/twitter list.png b/static/default/url_classes/twitter list.png
deleted file mode 100644
index 95e28236..00000000
Binary files a/static/default/url_classes/twitter list.png and /dev/null differ
diff --git a/static/default/url_classes/twitter syndication api collection.png b/static/default/url_classes/twitter syndication api collection.png
deleted file mode 100644
index b08d4f7a..00000000
Binary files a/static/default/url_classes/twitter syndication api collection.png and /dev/null differ
diff --git a/static/default/url_classes/twitter syndication api likes (user_id).png b/static/default/url_classes/twitter syndication api likes (user_id).png
deleted file mode 100644
index 720342ee..00000000
Binary files a/static/default/url_classes/twitter syndication api likes (user_id).png and /dev/null differ
diff --git a/static/default/url_classes/twitter syndication api likes.png b/static/default/url_classes/twitter syndication api likes.png
deleted file mode 100644
index 417f9ee1..00000000
Binary files a/static/default/url_classes/twitter syndication api likes.png and /dev/null differ
diff --git a/static/default/url_classes/twitter syndication api list (list_id).png b/static/default/url_classes/twitter syndication api list (list_id).png
deleted file mode 100644
index 983dd088..00000000
Binary files a/static/default/url_classes/twitter syndication api list (list_id).png and /dev/null differ
diff --git a/static/default/url_classes/twitter syndication api list (screen_name and slug).png b/static/default/url_classes/twitter syndication api list (screen_name and slug).png
deleted file mode 100644
index 8546e0c3..00000000
Binary files a/static/default/url_classes/twitter syndication api list (screen_name and slug).png and /dev/null differ
diff --git a/static/default/url_classes/twitter syndication api list (user_id and slug).png b/static/default/url_classes/twitter syndication api list (user_id and slug).png
deleted file mode 100644
index bff93d3d..00000000
Binary files a/static/default/url_classes/twitter syndication api list (user_id and slug).png and /dev/null differ
diff --git a/static/default/url_classes/twitter syndication api profile (user_id).png b/static/default/url_classes/twitter syndication api profile (user_id).png
deleted file mode 100644
index 4412799f..00000000
Binary files a/static/default/url_classes/twitter syndication api profile (user_id).png and /dev/null differ
diff --git a/static/default/url_classes/twitter syndication api profile.png b/static/default/url_classes/twitter syndication api profile.png
deleted file mode 100644
index 7914a875..00000000
Binary files a/static/default/url_classes/twitter syndication api profile.png and /dev/null differ
diff --git a/static/default/url_classes/twitter syndication api timeline-profile.png b/static/default/url_classes/twitter syndication api timeline-profile.png
new file mode 100644
index 00000000..de6a17ed
Binary files /dev/null and b/static/default/url_classes/twitter syndication api timeline-profile.png differ
diff --git a/static/default/url_classes/twitter syndication api tweet-result.png b/static/default/url_classes/twitter syndication api tweet-result.png
index f0431cfe..08094398 100644
Binary files a/static/default/url_classes/twitter syndication api tweet-result.png and b/static/default/url_classes/twitter syndication api tweet-result.png differ
diff --git a/static/default/url_classes/twitter syndication api tweet.png b/static/default/url_classes/twitter syndication api tweet.png
deleted file mode 100644
index b07b3855..00000000
Binary files a/static/default/url_classes/twitter syndication api tweet.png and /dev/null differ
diff --git a/static/default/url_classes/twitter tweet (i_web_status).png b/static/default/url_classes/twitter tweet (i_web_status).png
new file mode 100644
index 00000000..05d0522e
Binary files /dev/null and b/static/default/url_classes/twitter tweet (i_web_status).png differ
diff --git a/static/default/url_classes/twitter tweet.png b/static/default/url_classes/twitter tweet.png
index 4c72cf8a..37773efc 100644
Binary files a/static/default/url_classes/twitter tweet.png and b/static/default/url_classes/twitter tweet.png differ