changelog
-
+
version 447
+ - misc: +
- fixed drag and dropping multiple newline separated urls onto the client when those urls come from a generic text source +
- pages now cache their 'ordered' file id list. this speeds up several little jobs, but most importantly should reduce session save time for sessions with tens of thousands of files +
- common file resolutions such as 1920x1080 are now replaced in labels with '1080p' strings as already used in the duplicate system. also added 'vertical' variants of 720p, 1080p, and 4k +
- when a page preview viewer gets a call to clear its current media when it is not currently the page in view, it now recognises that properly. this was happening (a 'sticky' preview) on drag and drops that navigated and terminated on other pages +
- the various 'retry ignored' commands on downloaders now give an interstitial dialog where you can choose to retry 'all', '404s', or 'blacklisted' files only +
- manage tag siblings/parents now disables its import button until its data is loaded. imports that were clicked through before loading were being forgotten due to tangled logic, so for now I'll just disable the button! +
- reduced some more spiky database I/O overhead from the UI's perspective (now savepoints are performed after a result is returned, just like I recently did with transaction commit) +
- duplicate potentials search will now update the y in its x/y progress display if many files have been imported since the search was started and x becomes larger than y (due to y secretly growing) +
- fixed the default 'gelbooru md5' file lookup script. if you have a lookup script with this name, it will be updated to my new default automatically. I don't really like fixing this old system, but I am not sure when I will fit in my big rewrite that will merge it with the normal downloader system, so this is a quick fix for the meantime +
- if you are one of the users who had weird unfixable 404 update file problems with the PTR, please try unpausing and doing a metadata resync one more time this week. fingers crossed, this is fixed. please let me know how you get on too, fixed or not, and also if you have had 'malformed' database problems in the past +
- . +
- multi column lists: +
- improved the precision of longer text pixel_width->text and text->pixel_width calculations, which are particularly used in the multi-column list state saving system. another multi-column size calculation bug, where lists could grow by 1 character's width on >~60 character width columns on every dialog reopen, is now fixed +
- multi-column lists should now calculate last column width more precisely and accurately regardless of vertical scrollbar presence or recent show/hide +
- the snapping system that locks last column size to 5-character multiples can now snap up or down, increasing error tolerance +
- I added a hack to stop the bug some people had of multi-column lists suddenly growing wide, up to screen width, in a resize loop. I think it works, but as I cannot reproduce this error, please let me know how you get on. resizing the options->external programs panel seems to initiate it reliably for those users affected +
- . +
- profile mode: +
- all debug profile modes (callto, db, server, menu, pubsub, and ui) are now merged into one mode under help->debug +
- this new mode no longer spams popups, and it only prints 'slow' jobs to the profile log +
- it also makes a new profile log every time it is turned on, using mode start timestamp rather than client boot timestamp, and when profile mode is turned off, there is a popup summary of how many fast and slow jobs passed through during the log time +
- touched up profile code, timing thresholds, summary statements, and the help +
- . +
- special update rule this week: +
- due to the big file storage rework this week, there's some bit rot in older update routines. 447 cannot update databases older than 411, and it _may_ have trouble updating before 436. if this applies to you, the client will error out or warn you before continuing. I'd like to know what happens to you if you are v411-435 so I can refine these messages +
- . +
- boring database refactoring: +
- the primary current, deleted, pending, and petitioned files tables are now split according to service, much as I have done with mapping tables in the past. this saves a little space and accelerates many file calculations on large clients. if you have a client database script or patch that inspects 'current_files' or 'deleted_files', you'll now be looking at client_files_x etc.., where x is the service_id, and they obviously no longer have a service_id column +
- a new file storage database module manages these tables, and also some misc file deletion metadata +
- refactored all raw file storage updates, filters, and searches to the new module +
- the mappings and new file storage database modules are now responsible for various 'num files/mappings' metadata calculations +
- most file operations on smaller domains, typically trash or repository update files, will be significantly faster (since the much larger 'my files' table data isn't fattening the relevant indices, and worst case query planning is so much better) +
- cleaned up a ton of file domain filtering code as a result of all this +
- physical file deletion is now much faster when the client has many pending file uploads to a file repository or IPFS service +
- complicated duplicate file operations of many sorts should be a _little_ faster now, particularly on large clients +
- searching files with 'file import time' sort should be a little faster in many situations +
- tag repositories no longer bother going down to the database level to to see if they have any thumbnails to sync with +
- everyone also gets a local file id cache regen this week, it may take a few seconds on update +
version 446
- misc: diff --git a/help/reducing_lag.html b/help/reducing_lag.html index 5b1c3b7e..b6daf984 100644 --- a/help/reducing_lag.html +++ b/help/reducing_lag.html @@ -8,6 +8,7 @@
- The general steps to reproduce the problem (e.g. "Running system:numtags>4 is ridiculously slow on its own on 'all known tags'.") -
- Your operating system and its version (e.g. "Windows 8.1") -
- Your computer's general power (e.g. "A couple of years old. It runs most stuff ok.") +
- Your client's approximate overall size (e.g. "500k files, and it syncs to the PTR.")
- The type of hard drive you are running hydrus from. (e.g. "A 2TB 7200rpm drive that is 20% full. I regularly defrag it.")
- Any profiles you have collected.
-
+
hydrus is cpu and hdd hungry
The hydrus client manages a lot of complicated data and gives you a lot of power over it. To add millions of files and tags to its database, and then to perform difficult searches over that information, it needs to use a lot of CPU time and hard drive time--sometimes in small laggy blips, and occasionally in big 100% CPU chunks. I don't put training wheels or limiters on the software either, so if you search for 300,000 files, the client will try to fetch that many.
+Furthermore, I am just one unprofessional guy dealing with a lot of legacy code from when I was even worse at programming. I am always working to reduce lag and other inconveniences, and improve UI feedback when many things are going on, but there is still a lot for me to do.
In general, the client works best on snappy computers with low-latency hard drives where it does not have to constantly compete with other CPU- or HDD- heavy programs. Running hydrus on your games computer is no problem at all, but if you leave the client on all the time, then make sure under the options it is set not to do idle work while your CPU is busy, so your games can run freely. Similarly, if you run two clients on the same computer, you should have them set to work at different times, because if they both try to process 500,000 tags at once on the same hard drive, they will each slow to a crawl.
If you run on an HDD, keeping it defragged is very important, and good practice for all your programs anyway. Make sure you know what this is and that you do it.
maintenance and processing
@@ -16,22 +17,20 @@If you have an old, slower computer though, or if your hard drive is high latency, make sure these options are set for whatever is best for your situation. Turning off idle time completely is often helpful as some older computers are slow to even recognise--mid task--that you want to use the client again, or take too long to abandon a big task half way through. If you set your client to only do work on shutdown, then you can control exactly when that happens.
reducing search and general gui lag
Searching for tags via the autocomplete dropdown and searching for files in general can sometimes take a very long time. It depends on many things. In general, the more predicates (tags and system:something) you have active for a search, and the more specific they are, the faster it will be.
-You can also look at file->options->speed and memory, again especially if you have a slow computer. Increasing the autocomplete thresholds is very often helpful. You can even force autocompletes to only fetch results when you manually ask for them.
+You can also look at file->options->speed and memory. Increasing the autocomplete thresholds under tags->manage tag display and search is also often helpful. You can even force autocompletes to only fetch results when you manually ask for them.
Having lots of thumbnails open or downloads running can slow many things down. Check the 'pages' menu to see your current session weight. If it is about 50,000, or you have individual pages with more than 10,000 files or download URLs, try cutting down a bit.
finally - profiles
-Lots of my code remains unoptimised for certain situations. My development environment only has a few thousand images and a few million tags. As I write code, I am usually more concerned with getting it to work at all rather than getting it to work fast for every possible scenario. So, if something is running slow for you, but your computer is otherwise working fine, let me know and I can almost always speed it up.
-Let me know:
+Programming is all about re-editing your first, second, third drafts of an idea. You are always going back to old code and adding new features or making it work better. If something is running slow for you, I can almost always speed it up or at least improve the way it schedules that chunk of work.
+However figuring out exactly why something is running slow or holding up the UI is tricky and often gives an unexpected result. I can guess what might be running inefficiently from reports, but what I really need to be sure is a profile, which drills down into every function of a job, counting how many times they are called and timing how long they take. A profile for a single call looks like this.
+So, please let me know:
A profile is a large block of debug text that lets me know which parts of my code are running slow for you. A profile for a single call looks like this.
-It is very helpful to me to have a profile. You can generate one by going help->debug->xxx profile mode, which tells the client to generate profile information for every subsequent xxx request. This can be spammy, so don't leave it on for a very long time (you can turn it off by hitting the help menu entry again).
-For most problems, you probably want db profile mode.
-Turn on a profile mode, do the thing that runs slow for you (importing a file, fetching some tags, whatever), and then check your database folder (most likely install_dir/db) for a new 'client profile - DATE.log' file. This file will be filled with several sets of tables with timing information. Please send that whole file to me, or if it is too large, cut what seems important. It should not contain any personal information, but feel free to look through it.
+You can generate a profile by hitting help->debug->profile mode, which tells the client to generate profile information for almost all of its behind the scenes jobs. This can be spammy, so don't leave it on for a very long time (you can turn it off by hitting the help menu entry again).
+Turn on profile mode, do the thing that runs slow for you (importing a file, fetching some tags, whatever), and then check your database folder (most likely install_dir/db) for a new 'client profile - DATE.log' file. This file will be filled with several sets of tables with timing information. Please send that whole file to me, or if it is too large, cut what seems important. It should not contain any personal information, but feel free to look through it.
There are several ways to contact me.