Archive for November, 2005

A Good Day

Photoshop pushes a machine harder than most programs. It’s part of squeezing out as much performance from a machine as possible. Sometimes that means marginal hardware can show problems that look an awful lot like software bugs but aren’t. Or, even worse, cause a machine to mysteriously reboot (it’s happened). In other words, we take performance seriously, but that can sometimes tickle machine weaknesses.

So, as we’ve heard the occasional reports of mysterious pauses or bad slowdowns with CS2, we have really been paying attention. Trying to find a reproducible case can be very hard, though – and it’s hard to fix a performance issue if you can’t reproduce it (or at least have enough information to simulate it). And sometimes, through the course of dealing with a helpful user, it can become clear that there is something very weird or particular about the machine they have, and boy, at that point, do you really want to get your hands on that machine and take a good, deep look.

Well, through a series of events, we got lucky enough to find a user in the local area with a slowdown issue. So Seetha and I packed up his laptop in the car and headed out yesterday afternoon. When we got there, our user was so happy to have us.

We started with a demonstration of the problem. It took a while to happen. But when things started going “funny”, it was pretty clear that something was not normal. Ah, good, we thought. Not intermittent or one of those funny bugs that can simply disappear when two senior engineers walk in the room (you’d be surprised at how often *that* happens), but a repeatable problem that just happened to be tied to a specific machine.

I had us do the thing I always do when starting to try and figure out a bug – we pulled up Performance Monitor (on a Windows box – I pull up Activity Monitor on Macs, or install X Resource Monitor) and repeated the problem. Our user once again showed us the problem as Performance Monitor was watching CPU, disk activity, and free RAM. I have a set of colors I tend to use for these things, so that I can glance at Performance Monitor and know quickly what is going on. So we tried to reproduce the problem quickly. And the lines in Performance Monitor looked OK. So we sat back to think about what things might be going on – and after a short bit, I noticed that the blue line I use for free RAM was making a steady march downwards. With Photoshop idle. Ugh. And we watched the memory go to near-zero, at which point disk activity started going up, and we started poking around, and it now became clear why performance on this machine was getting so bad so quickly – a memory leak. A bad one. One that we had never seen before. So now the hunt was on for why…

We started up Task Manager and went to the Performance tab. Yup, available RAM was sitting at 3MB. On Windows XP, if your machine every has less than about 15MB of free RAM, you know you’re going to be in for trouble performance-wise. Ugh. So we quit Photoshop – which took a bit, since there was so much of Photoshop paged out due to the leak. I switch Task Manager to the Processes tab, added Virtual Memory Size, and started Photoshop, and didn’t touch it. Yup, there, plain as day. Munch, munch, munch went memory.

We did some first-step debugging – restarting without plug-ins, without scripting. No good. Still leaking memory. With all the easy answers gone, we hooked Seetha’s laptop up to our user’s network, installed the remote debugging stubs on the user’s machine, and started up a debug build. When the debug build started up, with all of it’s extra asserts and checks and paranoia built in, the smoking gun showed up right away. Aha. Bad font. And the light bulb went off in my head… The font preview menu. You see, I implemented the font preview menu to take absolutely zero extra time at startup. It generates all the previews on the fly, in the background, at idle time. Works really well, especially as most machines can read in a font and generate a preview just about as fast as they could read in a cache. We can usually generate all your previews in the first 20 seconds or so while the app is up, while some other programs that have WYSIWYG menus take more than twice as long to read in their cache.

Anyway, it matched the idle-time symptom of the memory leak we saw. And it struck me that if – **if** – a font failed in a certain way that caused our font rasterizer component to let the failure leak out (which isn’t supposed to happen) we could not only drop our memory on the floor but not remember that we tried that font. Bad news. So, we tried the workaround of turning off the preview menu, and of course, no memory leak. Now, both Seetha and I are eager to figure out which font it is, so we can bring the sucker back with us and dissect what went wrong. So, we bring up the debug build again, catch the problems, read the indicators, and get the font name. Cool. We turn on the preview menus and remove those fonts from the system, and start Photoshop and watch. Shoot. The memory leak is still happening. Seetha posits that it could be more than one font. So, we start a full-on search for the bad font (yup, the very same binary search-for-the-bad-font that we ask users to do when we suspect the problem is related to a bad font). Sure enough, with nothing but the minimal font set installed, Photoshop starts up fine – no leak. So we start moving fonts back, chunks at a time. No leak. We move more No leak. We’re back down to the one font we thought was bad, and moved it back. No leak. Huh?

At this point, we’ve actually completely solved the user’s problem with no workaround (all their fonts are back in place and the program is performing great). And we have enough information to simulate the bug back in the lab (which I’ve now done). So we pack up and head out, thanking our user for their time (I hope we conveyed how immensely we appreciated it – we really did). Our best guess is that some pre-caching or performance mechanism in the operating system somehow got a bad shadow copy of the font and that bad shadow copy was causing the issue, and that re-installing all the fonts (which is what we essentially ended up doing) force the bad shadow copy to be refreshed from the good copy. But that’s a guess.

Now, while I’d really like to know what actually went wrong, we ended the day with a happy user, information that might help some other users, and a cornered bug to be fixed for the next release. And both Seetha and I reflected on our way back to Adobe about all the bad software we deal with and programmers who wouldn’t even think of going to a user’s site to debug an issue (and certainly not 2 senior engineers like us).

A good day.

-Scott

What I did on my summer vacation…

OK, so it wasn’t exactly summer. But I recently had my third sabbatical, which I combined with vacation to make for a whole 6 weeks away from work. I highly recommend it. It’s why there haven’t really been any entries here for a while. I really made a break from work – heck, I didn’t even fire up my VPN client for the entire 6 weeks. Or check e-mail. No, I’m serious – I think in this age of blackberries and always-on connections that taking a time out is a very good thing to do. I had worked so hard in helping Photoshop CS2 get out the door that the “Jane, Get me off this crazy thing!” feeling was overwhelming. So I stepped off the merry-go-round for a bit.

Am I implying that I didn’t have any computer contact? Of course not. I spent some time fighting against my media center PC. There was *finally* an ATI video card driver update that allows ‘PowerStrip’:http://entechtaiwan.net/util/ps.shtm to set up a proper 1080i output on the analog VGA output. I had previously used the ‘Omega’:http://www.omegadrivers.net/ modified driver, which is cool, but doesn’t come through Windows Update, and sometimes has a few additional bugs. Anyway, the 1080i output to my old Mitsubishi HDTV is now much more predictable across sleep/wake and reboots. And nailing the refresh rate properly got ‘PowerDVD’:http://www.cyberlink.com/multi/products/main_1_ENU.html into a state where it’s just doing the upsampling and not doing any temporal funkiness. DVDs now play back really beautifully. Why not use the Media Center DVD player? It sucks – it’s inefficient at doing the upsampling, doesn’t reliably do 5.1 output over the optical out from the sound card, and it does some decoding on the display card without allowing for a software override. And the ATI display card totally suck in this regard – it’s got the ‘chroma upsampling bug’:http://www.hometheaterhifi.com/volume_8_2/dvd-benchmark-special-report-chroma-bug-4-2001.html on the red channel something awful! (You may not want to follow the link – if you have a DVD player with the bug, you probably haven’t noticed. Read the article and you will. And it will irritate you. I now can’t help seeing the bug on players when it’s there.)

I also tried to do some capture off of the TiVo. Unfortunately, this is where the z540 having suboptimal components hurt. It’s got a Hauppage capture card in it. And Media Center doesn’t have a “record from VCR/line input” option anywhere, and other capture solutions either try and do on-the-fly encoding (which always ends up with a glitch somewhere) or try and use the native card encoding, which in the Hauppages’ case is the HCW2 codec – which, of course, doesn’t have a software decoder for it anywhere. Yuch. The best I’ve been able to do is to a full-frame capture with a post-process encoding using ‘Windows Media Encoder’:http://www.microsoft.com/windows/windowsmedia/9series/encoder/default.aspx. Of course, Windows Media Encoder itself has some problems – its de-interlacing isn’t very good. And recording full frames takes a seriously fast drive – the “spare” firewire drive I had around can only really do 12MB/s sustained, which didn’t quite cut it (I wouldn’t mind ordering one of the HP Personal Media Drives that will slip into a slot on the front of the machine, but in typical HP fashion, there is no easy way to find out what sizes of those drives are available or what the real specs on the drives are). I tried some of the other capture programs: ‘Open Video Capture’:http://www.008soft.com/, which can’t do a proper full-frame capture; ‘McFunSoft Video Capture’:http://www.mcfunsoft.com/mvs/, which, beside the absolutely awful company name, froze any time I tried to have a full-frame buffer capture of any length, and which has a horrendously bad user interface – that always forgot all the settings any time you quit it.

I know what I want here, and that’s what’s frustrating. And Media Center should have done it – yet another component of it which is inadequate. I really want a video re-encoder that knows how to use capture card-native format (or full-frame) and re-encode that through the ‘DivX’:http://www.divx.com/ encoder – my favorite, with the best de-interlacing and really nice quality. Windows Media Encoder only does Windows Media Player format, which is unfortunate. The quality is reasonable, so I guess that’s what I’ll have to live with for now. Yes, if this was a build-my-own machine, I would have tossed that Hauppage capture card and gotten something that could output something useful. That’s the problem with the HP z540 (and probably with the follow-ons, though I don’t have direct experience with them) – you really don’t feel like you can replace any components. At least not without cutting up the back of the machine. And given it’s cooling problems, I’m loath to do anything that can make them worse.

Anyway, that’s certainly not all I did with my time off! It wasn’t even a vey significant part of it – just the part that generates the most comments because, well, when things go right you tend not to notice them, and dealing with that machine gives you plenty to notice.

I started off my time off with a couple of weeks on Kauai. We overlapped with my brother Douglas for a couple of days. If you’re in the Poi’pu area, and you see a guy in a red wheelchair, say hi. Kauai is my favorite of the islands – the pace there is definitely the slowest, and when on vacation, that’s pretty much what I’m looking for. Susan likes to travel more, but this time I just needed a vacation.

After that, it was mostly catching up on the to-do list. I washed windows – which, on our house, is a major undertaking. There are so many windows in our house that when we were calling around for estimates to replace them a few years back, we got asked more than once if this was a bid for an apartment complex. It’s not that big a house, just a lot of windows. And, as is my way with that sort of thing, if I’m going to do it, I do it right – got all the screens down and pressure washed, cleaned and the coated all the windows with a polymer coating that should keep them clean longer. Even got the window above the entry way that’s real hard to get to.

I refinished a chair. Nothing spectacular, just an old chair with lots of nooks and crannies, which makes stripping a bear. But then, the days were mild with no wind, so I was able to do the work out on the patio. Me, my iPod, and harsh chemicals – what more could you want?

I also dragged my bicycle down, cleaned it up and rode it for the first time in, what, 7 years? Ugh. Gonna keep it down now and squeeze some morning rides in on the weekends. Nothing like a ride in crisp fall air to clear the head.

Tuned up my 5-burner barbeque. It’s an older Turbo model – black enamel, serious looking (none of that pansy stainless steel here! :-). Replaced the starter on it, which was interesting – the contrast between the original and the new one showed years of design tweaking. I’m ready for Thanksgiving now. For the past few years I’ve brined and smoked a couple of smaller turkeys on it (15 lbs. each). I’ve gotten really good at it. Susan and I have a bunch of friends and family over on Friday (so that our friends aren’t torn between our gathering and their own family things). The past couple of years, the smoked turkeys have always managed to disappear the fastest.

Oh, and I fixed up the yard lights. Or at least did the bulk of the fixing up. After lots of digging and crawling under the house (which I hate – it’s tight under there and I’m not exactly small) and lots of cursing at the unknown persons who originally ran the lights (and thought it was OK to simply twist wires together, wrap them in electrical tape and bury them), I’m most of the way to having this project done. I was originally hoping to actually have a couple of days at the end of my time off for just pure bumming around, but, well, even things like the time it takes to dig holes and run wires can be mis-estimated.

So, it was nice to escape the complexity of thinking about Photoshop for a while. Like sore muscles, my brain just needed some resting up. Nice, physical tasks, out in the air, without much complexity to them. It’s about as opposite of working on Photoshop as I can imagine – just the sort of break I needed to prepare for the next round of digging in and fixing things up…