This is mine. I was meaning to do a Show HN post for it actually, just was waiting for the weekend :)
I've been using this internally for several months. For local deletes YMMV. There seem to be some sort of central lock in the NTFS driver, so the speed up comes from being faster with scanning and generally idling less. For freshly mounted fast drives you may see a speed up of 1.5-2.5 times, but in other cases it will be just 10-20%.
However for remote deletes this tool is really great. Due to the network latency, single-threaded deletes will spend most of their time idle - a request needs to get there and then back before the next one can be fired. But using threads allows to have multiple requests "in flight", forming a queue if needed, so the remote end is always doing something. A typical speed up for a LAN is around 2x, and for VPN links this should be even higher.
Thanks, appreciate the compliment, but there's not much to it really. Do version A, use it, change what's off or annoying, rinse and repeat. At some point it always converges to a stable point.
Pretty much the only criteria is to remove friction from fast paths and add some where safeguards are needed.
> The source is incomplete, because it depends on several libraries that are not a part of this distribution. However, with an exception of simple_work_queue, these libraries are simple Win32 API wrappers and their functionality should be fairly obvious from their usage.
Just curious - if they're trivial wrappers - why not share them?
It takes time to prep a public release. Review the sources, comment them, check formatting, add docs, legalese, build instructions, etc. I really don't like publishing stuff that's not in a proper order.
Second is that a public release needs to be maintained and this complicates our own workflow. We can, in theory, do a release and then abandon it, but it doesn't seem like a right thing to do.
First of all, great piece of software thanks for sharing it. Secondly re: releasing then abandoning some subset of code being “not the right thing”, I would submit that as a user I’d much prefer abandoned code than unreleased code. Please consider just slapping a disclaimer on it and releasing it.
will it delete files even if open in another process? this is my main bugbear with windows file deletion: it won't delete it and you need special tools to track down the process that has the file open
If they aren't opened in FILE_SHARE_DELETE mode, then they will in fact be locked against deletion... because that's exactly what file's current user wanted. Force-deleting a file in this case is not the wisest thing to do.
The underlying behavior is correct though - people with open handles have the priority unless they have explicitly stated that deleting a file while they use it is OK.
What tool? That's windows' behavior. I would agree that locking a file and preventing deletion is good or possibly queueing a file for deletion when it is free, but windows does not handle this scenario well.
I always assumed that a delete command would execute at max disk capacity on a single thread and that multi-threading it would even slow the command (more random accesses on the disk).
This utility would have been a godsend a couple months ago when I had millions of XML files to delete on one of our server (12 years of accumulation). It took several days (it's an old Windows 2003 server with SCSI disks I think).
It always amazed me that deleting a remote directory with thousands of files can be so much slower than remoting into the server, and deleting the directory. I don't want to recursively list the remote files and then delete them one by one. I just want to remove the directory from the remote filesystem!
Can a tool solve that problem? (I guess powershell remote scripts could?)
There's no way to delete multiple files with a single syscall. There's also no way to purge the folder of all its contents. At least not on Windows, using an API.
The syscalls are quick enough when done locally. What I think is going on is that for a directory with N files on a share, I get N (or likely multiples of N) network roundtrips.
If the network round trip time is 1second and the time to delete the directory with 1000 files is 1 second when done on the local drive, then I want the delete on the network share to be 2 seconds not 2000 or 4000 seconds.
For deleting - yes, you will get a round-trip per delete request. So if they are serialized (which is the case with standard tools), it will be excruciatingly slow indeed.
For scanning - you'll get the list of content in chunks. The size of a chunk depends on the size of the buffer you provide. For smaller folders it'll normally fit in a single request.
unlink(2) returns EISDIR (on Linux at least) if you try to unlink a directory. I'm not sure if you can still do it, but long ago you could unlink(2) a directory as root, but it was not a good idea since it did not recursively free up the contents back to the file system's raw storage.
To remove a directory on Linux you have to empty it out (recursively, using unlink(2) on files and rmdir(2) on empty directories) and then rmdir(2) the top level directory.
There's also "del /f /s /q" to delete just files, followed by "rmdir /q /s" to delete just folders; and there's also "robocopy /mir" with an empty folder as a source. From what I've seen straight "rmdir" is the fastest of these three.
Now I don't know is this makes things slower, but Windows File Explorer by default never deletes files. It just moves them to the recycle bin.
If you want to bypass that step you need to use the Shift + Del keys and for that scenario Windows even warns you the action you are request is a permanent delete.
This was my test actually. Because that is the exact reason that I would use this for.
And it was spectacular. I'm amazed at the speed at which it clears out an entire folder.
Another nice touch is if I ask it to delete a folder and that folder is open in a command prompt somewhere, it just goes ahead and deletes it's contents cleanly.
I've been using this internally for several months. For local deletes YMMV. There seem to be some sort of central lock in the NTFS driver, so the speed up comes from being faster with scanning and generally idling less. For freshly mounted fast drives you may see a speed up of 1.5-2.5 times, but in other cases it will be just 10-20%.
However for remote deletes this tool is really great. Due to the network latency, single-threaded deletes will spend most of their time idle - a request needs to get there and then back before the next one can be fired. But using threads allows to have multiple requests "in flight", forming a queue if needed, so the remote end is always doing something. A typical speed up for a LAN is around 2x, and for VPN links this should be even higher.
In any case, AMA.