Posts in Category "Performance"

Photoshop World – Heavy Lifting

Well, another Photoshop World come and gone.  This year Adam Jerugim and I did a presentation on setting up a machine for doing very large file work.  Thank you again to those who attended – we were up against some interesting stuff.  I know I promised to have this entry up on Monday, but I’ve been struggling with getting PowerPoint to let me extract the information in a good way.  I’ll probably have to update this entry a couple more times as I figure out how to get what I want pulled out.

I wanted to call out a couple of things that are currently buried in the speaker notes, and I’m not sure if I got them across appropriately.  First, setting Photoshop’s memory percentage to 100% only makes sense if you’ve got more than 4GB of RAM in the machine, and again, only if you haven’t run into trouble running the filters you need that way, and are on CS3.  We’ve improved our ability to back off in the case that the machine we’re on starts to page every version.  However, it’s still important to watch that free RAM (or, in the case of Vista, the amount still being used for the system file cache).  It’s important that you watch what’s going on on your system when pushing things to their limit.  If you’re regularly seeing free memory (or the amount of free memory + system cache on vista) go below 20MB, it’s time to back off that memory percentage setting and try again.

Someone had asked a question at the end about RAID types, and I wanted to repeat that here – for fastest performance, RAID 0 with 2-6 drives for the Photoshop primary scratch disk is the fastest way to go, but isn’t a good place for storing files unless you’re going to back them up very frequently (daily?) to a big server on the network.  RAID 0+1 or RAID 5 would add in the reliability at some cost in performance or the need for additional drives.  We still need to measure which of those is the best way to go.  It’ll be about throughput.

So, here’s the current version of the presentation.  The animations don’t come across, and I still have to go through the speaker notes and clean them up and get a good export of that.  Hopefully within the next day or so, and I’ll update this post.

psworldperformancepresentation_export

[Update: I meant to get the version of the presentation with extended notes up before my travels, apologies for not getting that done.  I’ll get that up and catch up on the comments when I return in October. -Scott]

[Update: Sorry this took so long, but PowerPoint and I had to come to an, um… understanding.  So below is the link to the annotated version of the presentation, with all slide builds manually exploded apart so that they are actually usefull. -Scott]

PSWorldPerformancePresentation_Expanded

Reap what you Measure

You reap what you measure.  Measure twice cut once.  Don’t tweak without measuring.


So you want Photoshop to perform faster?  You’re gonna want to measure.  But measure what?


There are many things on a system that can go wrong.  You can run out of RAM, a disk can be going bad, you can be bottlenecking on that disk, and you could, in fact, simply be trying to do too much at once.  Unless you watch most of this at once, you’re not going to know what could be going wrong.


Fortunately, both the Macintosh and Windows systems have tools available – built-in or easily available – which measure most of what you need to know.  On Windows, I use the built-in Performance Monitor.  On the Macintosh, there’s Activity Monitor, but I personally prefer a great little application called X Resource Graph, because I like the visual history of the strip chart format, and it has most of the indicators I like to use. 


On the Windows side, I have a standard set of things I like to map on each machine.  I use a consistent color scheme, so that I can go from machine to machine and just glance at Performance Monitor and know what I’m looking at – if you only have one machine, this may not be an issue for you.  I choose blue for memory, red for processor (and green for a dual processor system), yellow for networking items, and various of the dark colors for disk items, with a dashed style.  I map both Available KBytes and Available MBytes so I can play the dual-sensitivity trick – I adjust the scaling on Available MBytes so that the blue line is somewhere in the middle of the strip window, adjusting the scale on the graph so that the blue line ends up somewhere between 10 and 100 (on a machine with 1GB, you’ll end up with it somewhere in the upper half using 0.1, on a machine with 2GB to 5GB, you’ll end up with it in the bottom half using 0.01), then I adjust the scaling on Available KBytes to match, then set the scale to 0.001.  What this does is give you a gross-level idea of free memory on the system, and when memory gets below 100MB free (the danger zone), you get a detailed view.  For each physical disk, I like to add % Disk Time, I set the line style to dashed and a dark color, for no particular reason beyond that’s the way I’ve always done it.  And of course, mapping % Processor Time to a big bright red is to make it obvious.  I then hit the General tab and turn off all the Display Elements.  You may need the legend turned on until you’ve learned what all the colors are, but I’m used to this setup by now, and it lets me have a real uncluttered window with no wasted real estate.  While I’m there, I set the update interval to something useful like 5 seconds.  Yeah, it’s not as much fun to watch that way, but you can usually get a better sense of tends over time this way.


Once I’ve done that, I go back into the Performance Monitor part and use View / Customize View to turn off everything.  I then go into File / Options and set Console mode to User Mode, limited access, single window.  I then use File / Save As to save out perfmon.msc, putting it into my Documents and Settings\<USERID>\Start Menu\Programs\Startup folder, so that it loads up when I log in.  Yeah, you could save it somewhere else and open it when you need it, but I find that having it up all the time is a big advantage.  I then close Performance Monitor, and select perfmon.msc from the Start Menu, All Programs / Startup (you’ll see it there if you saved it there in the previous step).  That gets you this nice, clean, compact monitor window that you won’t mind always having up and that you can glance at (when it’s not buried behind the Photoshop main window – it’s a great window to go along with all the palettes on your new second monitor  and see what’s going on.


On the Macintosh side, things are a little easier.  You can go with the standard Activity Monitor, in which case I’ll leave it up to you to figure out where to watch, or you can install the excellent X Resource Graph, which will let you quickly map all the items you need, and again has a nice strip chart format that gets you that sense of history that can often help figure out what’s going on.  I know some of the folks around here like some of the other monitor programs, but when I glance over to see what’s going on, I don’t need to try and have me brain interpret fancy colored circles on anything.  The key thing here again is turning on enough items in the preferences, and set up a color scheme you like.  Again, I set this as one of my login items.  I really like X Resource Graph’s ability to stalactite/stalagmite some of the graphs, but I really wish I could have a strip graph of the memory items instead of the paging graph (well, in addition to, really).  And as on Windows, I like to slow down the refresh time to once every five seconds, to get a reasonable amount of history into those strip charts.  There are color schemes which you can download which aren’t so colorful, and won’t be so intrusive on your image editing, and I usually have mine set up a little more subdued than this.  And again, it’s a great thing to tuck over on that second monitor.


On both systems, the question becomes, just what are you looking for?


Well, let’s start with memory.  Photoshop uses a lot of memory.  And it has it’s own virtual memory systems because it has to deal with data sets far larger than the address space allowed to 32-bit applications (and we can’t just do a 64-bit version and it’s still not clear if it’s worth it), plus additional reasons I’ll cover some other time.  What it means is Photoshop is going to use the memory percentage of free RAM you’ve told it, and try and live within that amount.  What’s key here is that while living within that amount, you really, really don’t want the amount of system free memory to get below 20MB.  What happens if you get somewhere below that amount of free RAM (the exact amount shifts around between the operating systems, and there are other variables – 20MB seems to be a reasonable safe zone on both platforms) is that the operating systems start trying to get free RAM back by paging out some things.  Well, given that Photoshop at that point is probably occupying most of that RAM that the operating system wants to reclaim, it’s highly likely that parts of Photoshop memory will get paged out.  Well, when Photoshop then goes to try and operate on the part of the image that was in that memory that got paged out, it has to wait for it to get paged in.  Even worse, if the system paging file and Photoshop’s scratch file are on the same physical drive (meaning you only have one set of drive heads), what will often happen is that Photoshop wants to read a new tile into memory from it’s scratch files, but the memory it’s trying to read that into has been paged out – so a little gets paged in (operating systems have annoyingly small page sizes), Photoshop continues trying to read in it’s tile, a little gets read in, then more of the place it’s trying to read it into needs to get paged back in, then a little more gets read, then…  Well, you get the idea.  Now, when both Photoshop scratch and the paging file are on the same physical disk, each of those little flips between reading the scratch file and paging in the memory to read it into forces the drive head to go to that part of the disk.  Photoshop will now be running about the slowest it could run on your machine.    So, let me repeat myself.  You really, really don’t want free RAM to go below 20MB.  Complicating matters are the filters that like to allocate a single chunk of memory to hold the part that’s going to be operated on.  You’ll know these filters because memory will get used up in a big chunk.  Photoshop will try and be good about giving up tile memory to trade in, but it’s probably not getting it all back (I’ll cover that some other time, too).  If you use those filters often (or if you use a few other memory hungry applications at the same time), then your overall session performance may be better if you back off on the Photoshop memory percentage.  Again, the key is that free memory not go below 20MB (I think I heard that somewhere else before… ).


The next thing to watch for is disk.  There’s disk activity, of course, related to paging activity which will also show up in the paging graphs.  And then there is disk activity related to reading files, and related to reading scratch.  The latter is more interesting, especially as it relates to CPU usage.  Kick off a global filter on a big file, and depending on what the filter is, you’ll see a bunch of CPU activity, then a bunch of disk activity.  There should be very little paging activity during this period (you did watch that free memory amount, right?).  Shortening the period of time the CPUs are waiting for the disk would be the purpose of setting up a RAID array for scratch, if you’re so inclined.  You’ll notice that on a heavily fragmented disk, the disk read amounts won’t form a smooth graph.  To avoid scratch file fragmentation, setting up the Photoshop scratch on it’s own disk is a good thing to do. Concentrating on scratch over actual file open/save reads and writes is preferable because you are usually exercising the disk for scratch usage many times more than for file reading and writing.  It’s often instructive to watch how what you are doing in Photoshop affects the CPU and disk activity.


So, who else is hogging all that memory anyway?  That’s where Windows Task Manager or Activity Monitor can come in handy once in a while.  Simply bring them up (and switch to the process tab in Task Manager) and look for the other applications which are using big chunks of memory.  Sometimes, you’ll find surprises there, like e-mail programs taking up 80MB, or a web browser eating 40MB, or an MP3 player taking 60MB (yes, these amounts seem big to me – but I don’t know what’s going on inside of them, they could be being efficient, but I doubt it).  Now, if you run Photoshop occasionally, and you want to keep those other programs running, no problem – just turn back Photoshop’s memory percentage.  With that level of use, tuning Photoshop’s memory percentage to the last megabyte isn’t going to buy you enough over that short a term to justify the likelihood that you’ll get below that magic 20MB free number.  In the busy machine case like this, less is often more.  If a machine is used to run Photoshop, pretty much Photoshop, from morning to night, then watching and tuning the memory percentage is definitely going to save you some time over the long run.  And, even better, if there are unavoidable reasons why your machine acts slow sometimes (like a disk is already busy even before you start saving a file) you can catch it and plan for that inevitable coffee break.


Note that there are side issues, such as the current pause on Macs with more than 4GB of RAM in them (a stale leftover of the Unix days is kicking in and spoiling the fun – the disk cache that Photoshop writes the scratch file through in large memory situations is getting flushed occasionally, and the operating system ends up forcing Photoshop to wait until that’s done), but this should get you started with the gross-level tuning of your Photoshop environment that is fairly easy and, best of all, cheap.


-Scott

Shovelware

So, by this point we hopefully have all heard about spyware, adware, malware. And we all know that this is nasty stuff, and that we should avoid having it on our systems – they tend to slow things down, eat resources, get in the way, destabilize things, what have you. Bad stuff.

Today, I want to cover shovelware. Shovelware is that software which is of such low quality or is otherwise so nearly useless to you (or has an atrocious usefulness to resources ratio) that just about the only way the companies can get it in front of you is to pay hardware manufacturers to pre-install it on new machines, or otherwise bundle it with other software you do want. If you don’t get a choice to install it or not, it’s probably shovelware. Shovelware takes up resources, but usually isn’t as malevolent as spyware – it’s not trying to record keystrokes or the web pages you visit. But it does cost you. And often it ends up in front of you because someone somewhere along the supply chain didn’t have the right motivations – they weren’t thinking about how to help you, the customer, but how to extract the biggest revenue dollars. Maybe some engineer stopped being excited about going to work and it just became a job.

Who knows?

I’ve hit a bunch of examples of shovelware lately. It bugs me, because I like a neat, clean system. A clean system performs better, feels better. And logs in a whole bunch faster.

The first resource offense most shovelware commits is the task tray icon on Windows. Hey, some software really has a legitimate reason for starting up when I log in and having an icon there. Palm Hot Sync, a virus scanner, Microsoft AntiSpyware – these all have a legitimate reason for starting up when I log in, and might as well have an easy access icon. But QuickTime? Java? C’mon, these things can be lazy started. And don’t insult me by giving me an option to hide the darned icon. That’s not what I want – I want you to not eat resources just because I logged in. Some of these things are even more pointless. HP’s printer drivers now install an additional little tray icon utility, and I don’t see the value add – everything it gives you access to is available elsewhere as far as I can figure out. But can I stop it from loading at startup? Is some critical service wrapped up in that resource theft? Do I really need some WinCinema manager in the taskbar tray making login take an extra 10 seconds? And just why is it so hard to keep Microsoft Messenger from starting up?

Of course, there’s also the Start menu pollution and disk space munching that happens. I needed to burn a CD from an .iso image on my Media Center the other day, and the program I would normally use (CDRWin) doesn’t like that drive. So, I’d heard about Nero, decided to give that a try. First, the download was 100MB – crazy. Clearly, there’s other junk in there. When I installed the trial, it installed all sorts of shovelware I didn’t want (now, I don’t know if what they installed was good stuff or not – it was just useless *for me*, and that’s the key). I just wanted the burning component. But because I couldn’t just get and buy the burning component, I gave up, uninstalled Nero, and searched a little harder, finding CD Burner XP Pro. Not only was it free, it worked and didn’t come with any shovelware. Nice.

Heck, I think shovelware is one of the things that hurt RealPlayer so much. Once they started bundling all the shovelware with the player, I gave up on it. Who needs all that garbage – some of which verges on adware – when there are ways of getting media content without it?

Then there’s the shovelware that comes with a new system. Both Dell and HP are bad about this (I’m sure others are as well, I just don’t have direct experience with them). It’s why many of us have a policy of wiping out and installing from a *real* Windows XP install CD right after getting a new machine, and why I think most of those “restore” disks are practically useless. My policy is to never buy a machine that doesn’t come with a real Windows install CD – a policy I violated with my Media Center PC, and of course ended up regretting when I restored it to try and stabilize it only to find that all the shovelware HP was installing meant the system didn’t even start out too terribly stable. Yuck. Wouldn’t it be nice if a system just coming out of the box was actually *clean*, and not the mucked up, dirty, registry scrambled mess that most manufacturers think pass for acceptable these days?

Of course, there’s also the case of software that starts out useful and devolves. I like my MusicMatch radio, but over the past two years, the darned thing has gone from eating 10% of my machine CPU to just about half. I don’t know what you do to waste that sort of time, but that’s getting near the limit of tolerability – that is software that is on the verge of becoming shovelware for me (though I really like having music playing while I work, so I can tolerate quite a bit). MP3 players tend to skirt this edge more than other programs it seems, grabbing memory and keystrokes and otherwise doing things that get in the way.

Now, I’m sure that any and all this stuff may be useful to someone, somewhere. But it isn’t to me, and I don’t want to have to go digging around afterward shutting it off. I know conventional wisdom says that modern machines have enough power and memory for all this stuff, but there is just so much of this junk that it’s death by a thousand paper cuts. Every percentage of CPU actually counts – it could make the difference between a clean and glitchy video capture. And every chunk of RAM that is forcefully kept alive counts – when that blue line (free RAM) in my Performance Monitor hits 10MB, things are going to come to a crawl (I always have Performance Monitor running on Windows and X Resource Graph running on Mac, though Activity Monitor is OK in a pinch).

I suppose the worst part is that it’s just so darned hard to keep a machine clean. Yeah, I go into Add/Remove Programs regularly and clean up old junk – but sometimes things are bundled in ways that you can’t get rid of the bad without getting rid of the good. And even if you could, you still have to have a good registry cleaner because so many of the uninstallers are half-hearted efforts. And you shouldn’t really have to pull open the Program Files directory to go cleaning up disk files left behind, but often do. It used to be that you’d have to completely re-install Windows 98 every so often because the system itself became unstable. Now, while Windows XP itself is stable, it’s getting to the point where you have to do a re-install just to really get rid of all the vestiges of all that shovelware.

Sigh.

-Scott

A Good Day

Photoshop pushes a machine harder than most programs. It’s part of squeezing out as much performance from a machine as possible. Sometimes that means marginal hardware can show problems that look an awful lot like software bugs but aren’t. Or, even worse, cause a machine to mysteriously reboot (it’s happened). In other words, we take performance seriously, but that can sometimes tickle machine weaknesses.

So, as we’ve heard the occasional reports of mysterious pauses or bad slowdowns with CS2, we have really been paying attention. Trying to find a reproducible case can be very hard, though – and it’s hard to fix a performance issue if you can’t reproduce it (or at least have enough information to simulate it). And sometimes, through the course of dealing with a helpful user, it can become clear that there is something very weird or particular about the machine they have, and boy, at that point, do you really want to get your hands on that machine and take a good, deep look.

Well, through a series of events, we got lucky enough to find a user in the local area with a slowdown issue. So Seetha and I packed up his laptop in the car and headed out yesterday afternoon. When we got there, our user was so happy to have us.

We started with a demonstration of the problem. It took a while to happen. But when things started going “funny”, it was pretty clear that something was not normal. Ah, good, we thought. Not intermittent or one of those funny bugs that can simply disappear when two senior engineers walk in the room (you’d be surprised at how often *that* happens), but a repeatable problem that just happened to be tied to a specific machine.

I had us do the thing I always do when starting to try and figure out a bug – we pulled up Performance Monitor (on a Windows box – I pull up Activity Monitor on Macs, or install X Resource Monitor) and repeated the problem. Our user once again showed us the problem as Performance Monitor was watching CPU, disk activity, and free RAM. I have a set of colors I tend to use for these things, so that I can glance at Performance Monitor and know quickly what is going on. So we tried to reproduce the problem quickly. And the lines in Performance Monitor looked OK. So we sat back to think about what things might be going on – and after a short bit, I noticed that the blue line I use for free RAM was making a steady march downwards. With Photoshop idle. Ugh. And we watched the memory go to near-zero, at which point disk activity started going up, and we started poking around, and it now became clear why performance on this machine was getting so bad so quickly – a memory leak. A bad one. One that we had never seen before. So now the hunt was on for why…

We started up Task Manager and went to the Performance tab. Yup, available RAM was sitting at 3MB. On Windows XP, if your machine every has less than about 15MB of free RAM, you know you’re going to be in for trouble performance-wise. Ugh. So we quit Photoshop – which took a bit, since there was so much of Photoshop paged out due to the leak. I switch Task Manager to the Processes tab, added Virtual Memory Size, and started Photoshop, and didn’t touch it. Yup, there, plain as day. Munch, munch, munch went memory.

We did some first-step debugging – restarting without plug-ins, without scripting. No good. Still leaking memory. With all the easy answers gone, we hooked Seetha’s laptop up to our user’s network, installed the remote debugging stubs on the user’s machine, and started up a debug build. When the debug build started up, with all of it’s extra asserts and checks and paranoia built in, the smoking gun showed up right away. Aha. Bad font. And the light bulb went off in my head… The font preview menu. You see, I implemented the font preview menu to take absolutely zero extra time at startup. It generates all the previews on the fly, in the background, at idle time. Works really well, especially as most machines can read in a font and generate a preview just about as fast as they could read in a cache. We can usually generate all your previews in the first 20 seconds or so while the app is up, while some other programs that have WYSIWYG menus take more than twice as long to read in their cache.

Anyway, it matched the idle-time symptom of the memory leak we saw. And it struck me that if – **if** – a font failed in a certain way that caused our font rasterizer component to let the failure leak out (which isn’t supposed to happen) we could not only drop our memory on the floor but not remember that we tried that font. Bad news. So, we tried the workaround of turning off the preview menu, and of course, no memory leak. Now, both Seetha and I are eager to figure out which font it is, so we can bring the sucker back with us and dissect what went wrong. So, we bring up the debug build again, catch the problems, read the indicators, and get the font name. Cool. We turn on the preview menus and remove those fonts from the system, and start Photoshop and watch. Shoot. The memory leak is still happening. Seetha posits that it could be more than one font. So, we start a full-on search for the bad font (yup, the very same binary search-for-the-bad-font that we ask users to do when we suspect the problem is related to a bad font). Sure enough, with nothing but the minimal font set installed, Photoshop starts up fine – no leak. So we start moving fonts back, chunks at a time. No leak. We move more No leak. We’re back down to the one font we thought was bad, and moved it back. No leak. Huh?

At this point, we’ve actually completely solved the user’s problem with no workaround (all their fonts are back in place and the program is performing great). And we have enough information to simulate the bug back in the lab (which I’ve now done). So we pack up and head out, thanking our user for their time (I hope we conveyed how immensely we appreciated it – we really did). Our best guess is that some pre-caching or performance mechanism in the operating system somehow got a bad shadow copy of the font and that bad shadow copy was causing the issue, and that re-installing all the fonts (which is what we essentially ended up doing) force the bad shadow copy to be refreshed from the good copy. But that’s a guess.

Now, while I’d really like to know what actually went wrong, we ended the day with a happy user, information that might help some other users, and a cornered bug to be fixed for the next release. And both Seetha and I reflected on our way back to Adobe about all the bad software we deal with and programmers who wouldn’t even think of going to a user’s site to debug an issue (and certainly not 2 senior engineers like us).

A good day.

-Scott