Category Archives: Software Dev

Long Path Support

Support for long paths (ie paths greater than 255 characters) has been an increasingly requested feature and so I’m really pleased to see it added to the latest version of FileLocator Pro. Since FileLocator Pro does sometimes rely on the Windows shell for functionality such as file context menus it’s possible that, until Windows fully supports long paths, not every aspect will function perfectly but any issues should be the exception and not the norm.

File paths in Windows are more complicated than most end users realise but fortunately Windows hides much of the complexity, if you’re interested in some of the inner workings I recommend you read this blog post: DOS to NT: A Path’s Journey.

Although Microsoft has been adding long path support to Windows 10 it’s actually quite a difficult problem. Many third party applications have the value (commonly called MAX_PATH) hard coded since the Windows API documentation specifically refers to it. Therefore it’s potentially troublesome for Windows 10 to simply switch on long file support by default, those third party applications not expecting paths longer than MAX_PATH might start crashing with buffer overflows, or worse if the overflows are maliciously targeted.

Please try out the long path support in FileLocator Pro and, as always, if you find any problems please report it to Tech Support.

Performance

At the end of March there was a new update for FileLocator Pro titled March 28th Performance Release. The significant performance enhancement was concerned with how FileLocator Pro handles Boolean searches, especially when concerned with wildcards or word boundaries. It’s not such a big deal when searching for one or two keywords (modern processors are SO much faster than standard HDDs that the performance bottleneck is reading the data not searching it) but with faster SSDs (or cached data) and large keywords lists the performance improvements of the new Boolean searching routines really make a difference.

Keep checking the Version History page because there a plenty more upgrades on the way!

Agent Ransack 2012 (revisited)

Okay, you may have noticed a distinct lack of Agent Ranasck 2012 in, well, 2012. Unfortunately there was so much work going on with our other products that Agent Ransack work was pushed back.

However, work has started (I’ve even seen a demo) and I promise that it will be out in 2013. Honest. No excuses this time.

P.S. There’s also some really cool stuff in the pipeline for FileLocator Pro and FileLocator Network in 2013. Watch this space!

FileLocator Pro and Large Searches

FileLocator Pro Crash ReportDuring August 2012 we quietly added a new crash reporting module to FileLocator Pro. Based on CrashRpt (an open source product hosted on Google Code) it’s one of the most useful quality control features we’ve ever added, although we hope it’s a ‘feature’ most of our users will never have cause to see.

Since then you may have noticed an increase in memory management related upgrades to FileLocator Pro. It’s not a co-incidence.

We’ve had a slow trickle of crash reports over the last few months and while most were odd, quick to fix, edge-case samples the majority have been related to memory management issues. It didn’t take long to see that FileLocator Pro had a problem on low spec’d machines performing searches where the data was in the gigabyte range and involved millions of files. We found a few problems that were simply bugs in the code, e.g. algorithms that reserved more memory than was necessary, but some of the problems were more subtle function related issues.

By default FileLocator Pro will record up to 10,000 lines of text per file and each line can be up to around 20,000 characters. That’s not usually a problem when searching in a limited set of files. Rarely will a file have 10,000 hits or a line have 20,000 characters. However, when searching over a very large data set with criteria that might not be very selective (e.g. searching for the letter ‘a’ – which was the actual search phrase in one of the crash reports we received) it can be a problem. It can be compounded by searching through file types that may not have EOL (End Of Line) markers, such as EXE or DLLs. Finally to make the whole thing just a little bit trickier, what might be a problem on a scrawny 512MB laptop is not necessarily a problem on sturdy 16GB PC.

The trouble is that FileLocator Pro doesn’t know at the beginning of the search if it’ll find a few hundred files with hits on a few lines (easy), a couple of files with hits on 10,000 lines (not a problem) or a million files with each one reporting hits on 10,000 lines (problem… probably).

FileLocator Pro 6.5 introduces a pre-emptive based solution. Based on the amount of memory installed on the machine FileLocator Pro sets an upper limit for un-restricted results per search (from 20MB up to around 200MB). If during a search that limit is reached FileLocator Pro starts restricting the search. Results for each file are reduced to around 20 lines, with a maximum of 256 characters per line, and the restriction is retained until the search finishes. If the search still runs out of memory then rather than crashing, as it did previously, it terminates the search.

Our tests on very low powered machines with just 512MB have shown a huge improvement in stability for very large searches and so far we haven’t received any memory related crash reports. Job done? Not quite but it’s one more step in cementing FileLocator Pro’s place as the ultimate super fast, rock solid, search and data analysis tool.

In a previous post I talked about ‘pushing a button that I think does nothing’. I hope you can see from our response to these bug reports that when you ‘Push the Button’ and send us a crash report it most certainly does something!

PST and MSG attachment searching

In 2007 Joel Spolsky wrote a blog post about gnarly problems, called Where there’s muck, there’s brass. It basically argued that real benefit to consumers comes in solving gnarly problems not nice simple fun ones.

We’ve just had our own ‘mucky’ experience dealing with attachment searching in PST and MSG files. While the MSG format is nowhere near as complicated as the PST format both have nasty surprises when accessing the attachments.

However, once it was all up and running it was impossible not to have a silly grin watching a demo of FileLocator Pro finding some ‘secret’ text inside a PDF, attached to a MSG file, attached to an email in a PST file, that itself was zipped up and attached to an email in another PST file. How cool is that!

In ‘Other News’ we also have a new Q&A site. It’s the same sort of thing as StackOverflow but just for Mythicsoft products. Check it out: http://qa.mythicsoft.com

Where’s the Apple Mac version?

How would you answer the following question:

Do software developers write software for Windows because

a) Windows is REALLY cool and hip
b) Microsoft creates a great environment in which to write applications
c) 90% of all PC users use Windows
d) Microsoft understands and looks after 3rd party software developers
e) Windows users understand that paying for software provides much needed support for their favourite tools and utilities

My answer: “f) All of the above (except a)”.

What’s my point? Well, it’s a convoluted answer to numerous, very complimentary, requests to port FileLocator Pro to the Mac. Usually I reply something along the lines of “We don’t have the resources to support multiple platforms with minimal market share”. But it’s not as simple as that.

Continue reading

Automated Testing

Our automated test system has discovered bugs, often just before a big release, more times than I’d care to admit. It’s an invaluable tool and one recommended by most modern development methodologies. However, I’m not a fan of methodologies in general. Let me preface that with a little bit of personal history…

Back in 1994, while working for a bank in London, I was asked to come up with a specification standard that could be given to any old ‘monkey’ and would produce reliable results. Management were fed up with the code quality and productivity disparities through-out the development teams.

Continue reading