Java是一种更富表现力的语言吗？

Chet Haase's Blog

Chet is an architect in the Java Client Group at Sun Microsystems. He spends most of his time working on graphics and performance issues and futures.

Closures: Making Java a More Expressive Language?

Posted by chet on December 15, 2006 at 06:48 AM | Permalink | Comments (4)

I went to a talk by Neal Gafter at JavaPolis on Neal's proposal for closures in Java and saw some happy code. The syntax that Neal is currently proposing can sometimes result in a ":)", which, as Sun's John Rose noted, is a smiley in the code.

Let's call this emoticode: the ability for code to express itself emotionally.

Completely ignoring any relevant discussion about syntax options, semantics, or even closures themselves, I just wanted to voice my support for emoticode. Far too long have we labored with languages which have no inherent emotion. The characters just sit there passively in our editor, staring dully back at us, having nothing more to say than what the compiler tells them they can. Now, through new language syntax, we can see whole new language patterns developing where code can be much more expressive. Code can smile at us, or frown, or wink, or laugh, or do any of the many, wonderful things that emoticons can do in the trite emails and IM messages that we receive from friends.

Finally, we can finally have a meaningful relationship with the code we write.

Perhaps we need more language features that specifically target emoticode. Maybe we can flag bug patterns through emoticode syntax. Or we can encourage good programming practice through happy expressions. Beginning programmers may fall into patterns that use emoticode that laughs at their developer; this gentle poking fun can help these students learn better coding practices while encouraging them to develop a long and meaningful relationship with the code they write.

Don't just write solid code; write happy code.

All Wound Up

Posted by chet on November 30, 2006 at 08:30 AM | Permalink | Comments (4)

All I hear everyone talking about in the community these days is "Open Sores". Man, get a bandage for it and move on already.

Applet-JAX

Posted by chet on November 07, 2006 at 05:12 PM | Permalink | Comments (7)

Go to the Yapplet Demo

Historically, Java applet development has been somewhat of an exclusive game; you either use Java or you use some other technology.  In the AJAX world, this has led to either Java or a mix of other technologies.

But there is no reason why Java cannot be a player in this world.  Clearly, HTML is pretty good at doing text stuff, the DOM is handy for manipulating elements on the page, and JavaScript is a handy way to get at the DOM elements.  But Java has some unique capabilities, too (including performance, development tools and platforms, robust language and library features, yadda yadda yadda).  So how can we mix these worlds better, so that Java can be another tool in the web application toolbox?

Enter Applet-JAX (Hans Muller's awesome name for this approach); we explore some of the existing infrastructure that allows Java-JavaScript communication to work and thus allows Java to be yet another player in an AJAX application.

There are a host of ideas and possibilities here, but we'll start out pretty simple; go to the Yapplet Demo page to read about and run an applet demo that shows off some of what I'm talking about.

Quality and Testimonial

Posted by chet on October 25, 2006 at 03:19 PM | Permalink | Comments (0)

Have you played with Java SE 6 yet? Do you like what you've seen? Do you want to dance the dance with fame by providing us with a testimonial about what you liked? Please contact our marketing folks at [email protected] and tell them a story about what you've enjoyed about the release. They may ask for some sort of bio and picture (that's where the fame part really kicks in).

Java on Vista: Yes, it Works

Posted by chet on October 06, 2006 at 02:23 PM | Permalink | Comments (29)

The Rumor

Hey, I like a rumor as much as anybody (Hey, did you hear that Larry Ellison is acquiring Europe?). But when the rumor is clearly wrong and it involves a project that I'm actually working on, I feel obliged to correct the misinformation (as did Java 2D's Dmitri Trembovetski, who has responded to the relevant threads in the next paragraph.  Go Dmitri!).

There was a post on Microsoft Watch, Windows Vista: Aero Glass and Java Don't Mix, last week that claimed that Java does not work on Windows Vista.  OSNews also picked it up. Then Javalobby picked it up on a forum entitled "Running Java on Vista Disabled Aero/Glass UI Effects". Imagine my surprise when I received the Javalobby newsletter with the subject "Java and Vista Not Playing Well Together".

I especially love this quote on Microsoft Watch:

Sun Microsystems would do well to give a ring to one of the interop contacts at Microsoft that came out of the firms' historic make-nice agreement back in 2004, and figure out how to make Java apps first-class Vista citizens.

Thanks for the tip, Jason; I'll pick up that phone right now...

So: Is the rumor true? Did Sun actually forget about Vista ("Whoops!") completely?

The Truth

Nope. The rumor is not true; we actually work great on Vista. (The rumor about Larry, on the other hand...).  In fact, we have been tuned into this release and making Java work on it since it was named after a breed of cattle.  Between regular calls with Microsoft, interaction with their engineers when problems or questions arose, and regular testing and engineering during Vista's development, we have been building a rock-solid release of Java for Vista.

So how did this bizarre rumor originate? Well, older versions of Java do have problems on Vista, and that's what the original report was about; someone tried running some older version of Java on Vista and noted some problems.  But that's like saying that your favorite XBox game, Bloody Mess X, doesn't work on XBox360.  Of course it doesn't; the original game was written for a completely different system.  Why should you expect it to work out of the [x]box on this new platform?  Presumably, given the popularity of Bloody Mess X, the developers are working on a port of the game to the new console ("Even more blood!"), so you will still be able to run your favorite game on this new system.

The port is the key; the new system is different enough in its fundamentals that software for the older system would not just work, but had to be ported.  It would be great if your game just worked.  Just like it would be great if Java just happened to work on Vista with no changes. But it just didn't happen that way; the platforms are different enough that changes are necessary.

It's the same thing here; Vista is not just XP++; there are fundamentally new things about the system that makes older software break. Is all software broken?  Probably not.  But the more of the system an application uses, the more likely it is to run into issues where the system has changed, and need to react to those issues.  In our case, Java is not just a simple win32 GUI application; it is a runtime platform with deep rooted needs in the operating system, the networking stack, the security model, the graphics system, ... if any of these change significantly, then we need to change our software in reaction. And in the case of Vista, it has been an ongoing process of learning, testing, debugging, submitting bugs against Microsoft, fixing our bugs, re-testing, ....  And since Vista has been a moving platform during the Java SE 6 development process, we've been in this development cycle continually with every new drop of Vista (they are still releasing weekly builds for us to test; we just found a bug in RC1 that has since been fixed in the latest release we got yesterday).

But the extra hassles of debugging a moving platform does not mean that we are unaware of the issues and haven't actually been fixing them; Vista has been one of the highest priorities for our Java SE 6 work.  In fact, Java SE 6 has been working quite well on Vista for months.  Some of the more obvious bugs (like Java 2D's disabling the Aero Glass whizzy desktop management system) have been fixed and available in snapshot releases of Java SE 6 for many months. And as we fixed problems in Java SE 6 (our primary platform for Vista support), we have also been backporting the more important fixes to older releases so that we can provide updates to these releases for customers that require the older releases to work on Vista as well.

As pictures are worth between 999 and 1,001 words, here are a few to help you see where we are at.  These are screenshots of Java GUI applications running on the latest versions of Vista (two of the SwingSet2 demo, one of Azureus, and one of Swing-based NetBeans 6):

 

 

 

The Problems We Fixed

Since this is a tech blog (okay, apart from the occasional geeky humor tangents, like this, this, this, and this) and not a whitepaper, I get to dive into a little more detail than I would otherwise.  In a whitepaper, we might say something like "We work well on Vista" and then defer to a PR representative whom you can call to have them say the same thing (with a lot more words).  But I'm thinking that Java developers might actually be interested in some of the technical details that we had to deal with to make our stuff work on Vista.  Those looking for things to occupy their time can chase down the real details in our bug database, of course, but I'll hit some of the highlights here:

Aero Glass

The "Aero Glass" desktop is enabled by the Desktop Window Manager (DWM) of Vista.  This system runs the desktop graphics on top of Direct3D.  All "windows" on the desktop are actually texture maps which are composited together onto the screen.  This system enables various effects that you can see on the screen, including: translucent title bars, application previews in the task bar, and the new Alt-Tab and Cmd-Tab functionality for switching between applications.

Problem 1: DWM disabling

• Symptom: On older releases of Java (1.4.2, 1.5 prior to update 8, SE 6 snapshots before about April of 2006), running a Java application would "punt the DWM", or would effectively disable the Aero Glass effect.  When you launch Java, you will see a change in the desktop appearance, the window borders will turn opaque, you will lose some of the whizzy effects, and you will essentially get more of a XP look & feel for your application windows.
• Problem: Java 2D uses DirectX for creating the Swing back buffer (and other offscreen images) and for copying these images to the screen.  Part of our sanity-check that the DirectX system is functional is to verify whether we can "lock" the screen, which gives us the capability to write or read pixels directly from the screen (we actually no longer use this facility for onscreen pixels; locking the screen to do this ends up causing artifacts with GDI graphics like software cursors and fading menu effects).  Locking the screen is also the easiest way (unfortunately) to flush the current queue of graphics commands (something that is necessary when someone calls Toolkit.sync(), for example). Locking is also a good way to just sanity-check that things are functional with DirectX before we start depending on it.  But locking is also, unfortunately, the quickest way to make the DWM punt on Vista; because of the new composited-desktop approach, applications in DWM no longer have direct access to the screen itself.  If an application demands access (such as through this older DirectDraw interface), Vista will oblige the request, but only by first disabling DWM, and thus disabling Aero Glass.
• Solution:  We found less obvious ways of doing what we needed (sanity-checking DirectDraw as well as flushing the graphics queue) and no longer lock the screen.  This avoids the DWM punt and makes Java run much more seamlessly on Vista.

Problem 2: Graphics artifacts (black rectangles)

• Symptom: Once we had fixed the DWM-punting bug above, we unmasked the problem that Swing windows would sometimes be littered with black boxes or draw the content in the wrong place.  The problems were intermittent and depended somewhat on the particular graphics operations that were taking place (scrolling or moving JInternalFrames around seemed particularly problematic).
• Problem : Desktop Java ends up using both DirectX and GDI to draw content into a Java window.  We use GDI for basic windowing functions such as creating the window and other heavyweight components that Swing uses underneath.  We use DirectX to get some of the more advanced 2D capabilities that Swing benefits from, such as caching the back buffer in video memory for accelerated copies to the screen. This worked fine on Windows pre-Vista ; GDI would draw directly to the screen, Direct X would copy directly to the screen, and life was good.  But in the new composited-desktop world of Vista, this system breaks down.  GDI is now rendered in software into a GDI texture map, DirectX is rendered into a different texture map, and, like small kids crammed together into the back seat on a long car trip, they simply don't play well together. Our use of both rendering systems for the same windows simply causes mayhem in the Vista rendering system and artifacts on the screen. The response we got from Microsoft when we pointed out the issue was akin to "Ohhhhhh - you're doing that?  That'll never work". See Greg Schechter's blog on the topic for more details.
• Solution: The quickest solution was to simply disable our use of DirectX on Vista.  We are basically forced to use one or the other: GDI or DirectX.  We cannot avoid GDI because we need access to basic window system capabilities, so DirectX had to go. There is a more complete solution in the works (for Java SE 7) that will involve a much more complete implementation on top of Direct3D with a more involved and capable buffering system that will allow us to use DirectX in a much more Vista-friendly manner.  But until we have finished that work (how is it going, Dmitri?) we will be using a software-rendering approach on Vista.  The graphics performance geek in me is not overjoyed about the solution; of course I want to see as much hardware acceleration used as possible. All of the time. For everything. But in reality, typical Vista machines (which will be pretty beefy) running typical Swing applications (with basic, static GUIs) will run absolutely fine.  In fact, given our basic level of acceleration so far, many current systems with decent CPUs (not just the high-end ones) run typical Swing applications faster than they would with this level of acceleration; it all depends on the combination of system resources and application requirements.

Swing Native Look & Feel

Vista has a completely new look & feel.  Beyond the Aero Glass effects discussed above, there are simply new themes and new looks to the buttons, the windows, and all of the other normal desktop widgets.  But Vista did not simply tweak the XP look & feel resources; they implemented these new looks through a completely different mechanism.  And they added animation to some of these widgets.  Swing's old approach of using the XP look & feel resources had to be re-thought in the context of this new world.

Problem 1: New native look & feel

• Symptom: A Swing application using the native look & feel on older Java releases actually does look native on Vista; it just happens to look like a native Windows 2000 application.
• Problem: In Windows XP, Swing derived its native look & feel from resources on the system which XP itself used.  So, essentially, we were using the same core resources as XP itself to make our components look native.  However, there were some corner cases in which this did not work on XP.  Even more problematic, Vista uses a completely new system with new resource locations and formats.  So not only did we not look Vista-native on Vista; we didn't even look XP-native (because the XP resources no longer exist).
• Solution: In Java SE 6, the Swing native look & feel for both Windows and GTK was re-written to use the native platform rendering engine to draw Swing widgets.  Instead of our old approach of using the native resources and doing our own rendering, we actually call the native rendering system to draw the widgets for us (on Windows, this is done through the UXTheme API). This means that we are finally able to achieve true native fidelity for these components because, by definition, our components are drawn in the same way that the native components are.  Not only did this fix corner cases for our XP look & feel, we also made our native look & feel work successfully on Vista.  More information about this work can be found on Bino George's and my blogs. The fix has been available in Java SE 6 snapshots since as early as build 14 (for reference, we're now past build 100).  The work has also been backported to J2SE 1.5, and is available in the currently available 1.5 update.  There have been ongoing fixes to other minor issues in native look & feel since then (Vista draws some components in new ways, which we needed to incorporate into our implementation).

Problem 2: Native look & feel animation

• Symptom: Some Vista components animate between different states. For example, moving the mouse over a button will cause a cross fade between the default state and the mouseover state.  Meanwhile, Swing components (like pre-Vista Windows native desktop components) have always simply switched between these states.  So where you might see an animation on a component in a native Vista application, you would see a simple (non-animated) state change in Swing.
• Problem: There was no animation system on older operating systems; components simply changed state and would be drawn in the appropriate way for that new state. In order to handle this change in behavior, we had to implement new logic to change the appearance over time.
• Solution: Igor Kushnirsky implemented Swing animation support in build 97 of Java SE 6 (the latest snapshot release on the download site has these fixes).  There are still some cases not completely handled (for example: progress bars have an animating "sparkle" every second or so that we do not currently render), but things look good in general.  Further fixes are planned for Java SE 6 update releases as well as Java SE 7.

Deployment

One of the more major areas of change in Vista affected the deployment team.  Vista has made some significant changes in its security system, in an attempt to make it more difficult for malicious software to gain control on the machine's resources.  For example, the user is now prompted when an application needs access to a system resource such as the registry; they will get a dialog asking whether they approve this access, and asking them for the Administrator password if they are not logged in as the administrator already.  Internet Explorer 7 (IE7) takes this a step further and protects that entire process from accessing the raw system, so that even if an application inside the browser gains access to the system, it can only perform operations inside the very restricted sandbox that the browser offers.

These changes had wide-reaching implications for our various deployment pieces, including the Installer, Java Update, Java Plug-in, and Java Web Start.  Many of the issues were overlapping, so I'll just cover the general areas of problems that we had to overcome.

Problem 1: Administrative Privileges Required

There were many cases in which we need administrator privileges in order to perform a necessary deployment task.  This includes storing things to registry keys, writing to locations on disk, and executing processes.

• Symptom: Some installer/uninstaller situations would not work correctly, some things that were formerly unknown to the user would now be more visible (for example, Java Update would ask at every login whether you wanted to let it run), and some things would need to ask your permission to write to a location that would not normally be available to a process.
• Problem: Vista tries to lock down system access as much as possible.  Some processes (such as installers) need "elevated privileges" that require user permissions to run.  In some situations, we would run an installer as an exec'd process, which would fail because there was no opportunity to ask the user for appropriate permission.  Similarly, Java Update would need elevated privileges in order to check for an update; since update runs at every login, this would happen whenever the user logged in (which, frankly, gets pretty annoying).
• Solution: We figured out ways to run the installer without resorting to exec'ing processes, so those problems were fixed.  There is no easy answer to the Java Update problem, so we simply reduced the frequency at which Update runs.

Problem 2: IE7 Sandbox

IE7 on Vista is really clamping down on security holes, but restricting access to the filesystem from the browser process and everything associated with it.  This means that Java Plug-in, which runs as an ActiveX control within IE7, is wholly contained within that IE7 sandbox.  This has various implications, from the ability to save into arbitrary locations to the ability to read from and write to the normal deployment installation directory (which Java Plug-in shares with Java Web Start).

• Symptom: Java Plug-in was unable to cache jarfiles and share other data with Java Web Start and the Java Control Panel. 
• Problem : Java Web Start and Java Control Panel run as standalone processes, and therefore have no trouble reading and writing to necessary places on disk (such as the usual deployment directory).  Java Plug-in, however, could not access these directories, and would thus end up writing to a new virtualized location on disk which could not be seen by these other applications.  We could no longer share the information between these very related applications. It was also not clear whether the virtualized directory used by Plug-in would actually be persistent; there was no guarantee that these virtualized locations were permanent.
• Solution: Later releases of Vista provide a persistent directory that all processes can reach.  The directory is within the sandboxed area that the IE7 browser can write to, so the overall filesystem is still protected, but as long as other non-IE7 processes (such as Java Web Start and the Java Control Panel) can reach this location, we have the sharing we need.  We chose this directory to house our deployment directory so that all of our Java deployment components could use and share it effectively.

Problem 2a: Vista and IE7 Sandboxes: The Unfixable

There is an issue worth noting here than cannot be fixed: IE7 on Vista will not let us access the file system outside the IE7 sandbox for applets.  This means that even "signed" applets have no permission to write to arbitrary locations on disk.  This behavior differs from the old behavior of applets on Windows, but given the restrictions of IE7, there is really no way around the problem.  Applications will need to adjust and write to locations that are reachable from their applications.  Note that if an applet simply needs to save information and access it later, this is completely possible; the information will reside in the sandboxed area of the browser.  But if they want to save information in a place that is accessible by other applications, that is trickier since applets cannot save outside of this area and the area may not be obvious to applications outside the browser.

There is another related issue also worth noting: Vista itself has proclaimed various directories off-limits for writing.  For example, it is no longer possible (or at least no longer trivial) to save files to the root directory (C:/) or to other system-level folders; non-administrator users are typically restricted to writing only within their home directories.  This change will affect not only applets, like the IE7 sandbox constraints noted above, but also standalone and Java Web Start applications.  This is obviously not a Java-specific problem, but is one that our users must deal with just like they will have to deal with it in other native applications; files must be saved in Vista-friendly locations.

Problem Summary

That's it for the techy details.  Hopefully you found it interesting; my purpose in detailing this stuff was not to bore or frighten you with the details, but rather to interest you in the relatively huge changes that Vista required from us, and to show why simply running out of the box on Vista was not really feasible for such a large and diverse platform as Java.  But, again, we did the work, we work well, and we encourage you take the time to run Java SE 6 on Vista to see how well it all works together for your applications.

The Plan

Many people reading this might be wondering about things beyond the bugs we fixed, for example: "When can I get a release of Java that works well on Vista?".  Well, here is the plan.

Java SE 6 is the Best Solution for Vista

First of all, you should note that the primary delivery of Java for Vista is Java SE 6; that release has received most of our focus during the Vista beta release timeframe, and it is where most of the fixes to the known problems currently reside.  We are just finishing up that release and it should be done and shipping by sometime next month.*

In the meantime, we encourage you to go to the Java SE 6 download site and get the latest snapshot for testing; the release is pretty close to final, so it is working very well at this point. In particular, all of the serious Vista problems have been fixed in this release for months, so it is a particularly good test vehicle for Java on Vista.

It is also worth mentioning that we are still aggressively pursuing OEM deals. We have distribution agreements with over 20 PC manufacturers, including all the top 10. They have all been helping us test Java SE 6 as they prepare their new lines of Vista-based systems for shipment, so that Java SE 6 will just be there on any of their new systems running Vista.  By the way; if you have a local PC maker who you prefer over the big guys (or if that is your business), let them know they can sign up to bundle Sun's Java Runtime on their systems, just like Dell, HP, Sony, and all the others do just by going to http://java.com/pcoem.

J2SE 1.5 Will Also Work

Many of the Vista fixes have already been, or will soon be, back-ported to J2SE 1.5.  However, don't look for everything to work exactly the same; our primary focus was to make Java SE 6 the main release vehicle for Vista. In an ideal world with infinite engineers working on everything, perhaps we would make sure everything worked the same everywhere.  But here in the Real World, we have to pick our battles, and the battle we picked this time was to make the new release the primary one for Vista.  Besides the effort of simply moving code back and forth between releases, there are architectural differences between some areas of the current and older releases that makes porting efforts more difficult and time-intensive, so we carefully choose what to back-port, based on a cost-benefit tradeoff. 

J2SE 1.5 should work fine, but there may be some nuances that may not be as perfect.  For example, the basic Swing native look & feel work has been back-ported, so that Swing native look & feel applications look like native applications on Vista; however, some additional Vista-specific fixes in this area (such as component animation) may not be back-ported, so the fidelity may not be as close as that in Java SE 6.

Some of the Vista fixes are already in the current release of J2SE 1.5 (as of this writing, update 9 is available on the java.sun.com download site ).  For example, the "Aero Glass" issues described above were addressed way back in update 8.  But the full gamut of Vista work that we feel is necessary for J2SE 1.5 should be available in update 11, which we hope to release around January of 2007.*

J2SE 1.4.2 Will Basically Work...

We plan to also backport necessary fixes to 1.4.2.  However, the same caveat goes for 1.4.2 that I detailed for 1.5, but even more so; we plan to fix only the necessary items to make 1.4.2 work on Vista, not to spend much time improving upon the basics.  A good example is native look & feel.  The rearchitecture of Swing's native look & feel engine involves a lot of code; we do not feel that it is worth the time and risk porting this code to this older platform with Java SE 6 being the primary release vehicle.  We see 1.4.2 as being functional, usable, and perfect for situations where a customer is absolutely locked into that particular release for now.  But we encourage developers and customers to migrate to a more full-feature Vista release soon.

We plan to ship this Vista-enabled update of 1.4.2 sometime following the Vista release of J2SE 1.5. I hope this will be in the first quarter of 2007*, but details are still being nailed down.  Our primary focus now is on finishing Java SE 6 and making it a stellar release, then jumping immediately onto the J2SE 1.5 update to make sure that it has the Vista support it needs.  Then when we are satisfied that these vehicles are robust, we can address Vista on J2SE 1.4.2.

* Don't quote me too exactly on dates.  First of all, I'm an engineer.  You should really talk to a business person that does this kind of stuff for a living if you want official dates and commitments.  These dates are pretty solid based on our schedules and projections, but you're a software person (aren't you?  how'd you get to this blog otherwise?) and understand that software release dates, like flies on your lunch, can move quickly as you get close to them.

Get the [arti]facts: Browser image scaling

Posted by chet on October 02, 2006 at 08:03 PM | Permalink | Comments (8)

I've been playing around with various web and desktop technologies lately, and noticed a funny thing when passing images back and forth between the browser and an applet: IE7 thinks my image is a different size than it actually is.

Here is the scenario: I have an image on my web page (a URL in an tag) that I'd like to hand off to an applet.  I want the applet to be sized appropriately, so I set the applet width/height according to the size of the image.  Then I display the image in the applet ... and the image is smaller than the applet.

Let's try that again.  I have an image that is 100x100, I create an applet of size 100x100, I use a simple drawImage(img, 0, 0, null) call in the applet, and I have a significant border to the right and bottom of the image; the image is not taking up the entire size of the applet, even though I specifically created the applet to be the same size as the image.  Also, the image displayed in the applet is clearly smaller than the image displayed through the tag in HTML.

Then I run it in Firefox and it works the way I thought it should to begin with.

I pounded my head against this for a while, and finally debugged the problem to my "DPI setting" on my Windows XP box.

Apparently, IE7 (and possibly IE6, although I no longer have it available on my system) auto-scales images in the browser according to your current DPI setting.  It considers 96 DPI the default, where one physical pixel on the screen equals one pixel in the image (so an image won't be scaled).  But any other DPI setting will cause auto-scaling.  On my system, a laptop with a relatively dense 1400x1050 screen, the setting is 120 DPI, or 125% of the default.  The net effect is that images will be scaled to 125% of their physical size when displayed in IE7 on my laptop.

Here is a screenshot that show the problem.  Ideally I would use PNG, a nice lossless image format, but IE's support for PNGs is somewhat broken; hopefully this JPEG will do the trick (if none of the images look right, save the image locally and open it up in a different image viewer):

 

The image I used is a simple black-line grid on a white background, where the black lines and white pixels are spaced very regularly away from each other (always one pixel of separation).  With this simple image, it is easy to see the scaling artifacts that are bugging me.

What I wanted was that the images on the first line would all look good. If I simply load an image with no width/height specified, it should create a space of exactly the right size.  Similarly, if I specify a width/height that matches the size of the image (I tried this with both "100" and "100px" to see if it mattered), then I should get an image area of the right size. 

What I got was an image area that was clearly not the right size; the distortion you see in these top three images shows an irregular grouping of black/white pixels, which shows that the image is not being mapped one-to-one to the onscreen pixels.

To test the theory further, I loaded the images in the second row with width/height specifications of "80" and "80px".  Magically, these images look perfect on my system.  I pre-scaled the images to account for the 125% scaling that I now knew IE7 was going to do.

Now, if we look at what happens on Firefox, we see the reverse (again, if you're not seeing what I'm describing, save the image and view it in a different application):

 

Firefox shows the first three images all looking good, and the bottom two images distorted. Firefox is clearly not auto-scaling the images, and when I pre-scale the images, it ends up causing distortion for Firefox where it fixed the problem for IE7.

Why Should We Care?

For the most part, images are being viewed in the browser and these kinds of scaling differences are probably irrelevant in that context; you want your photos to look nice, the browser does a decent job of scaling things, and the photos look fine. 

So what's the problem?

The problem is when we need to actually access the pixel information of these images, we're going to need finer control.  And if the browser is hiding that information from us, the browser makes a poor UI engine.

For example, if I need to edit a specific area of pixel data on an image, I'm going to need to find out exactly where those pixels are.  Or if I want to work on individual pixels, I don't want to be working on some pre-scaled version of the image, but rather the raw image data.

But here's a better example, and one that you might have already hit: Google Maps.

I've been using Google Maps off and on since it was introduced; it's so nice to have a dynamic mapping interface instead of the old batch retrieval system of the original mapping sites.  But most of the time, the "Directions" results of Google were just ridiculous.  That is, the directions themselves were correct, but the path drawn on the map was way off.  I don't mean that it had me taking the wrong highway to work, but that it had me driving through the middle of the bay (where there ain't no bridge). 

Maybe Google Maps just had it in for me, but I don't think so.  When I realized what was going on with my images, something clicked in the cobwebs of my memory and I thought I'd see if Google was failing for me for similar reasons.

First, I needed a nice test.  The roads around here (the SF bay area) are way too non-linear; it's hard to do easy comparisons.  So I did a search around Northfield, Minnesota.  If there's one place in the world that knows its cardinal directions, it's the American midwest.  (Aside: My Iowan grandfather once gave directions like so: "If you're coming from the North, then it'll be to the East".  And if we came from the South?).

I ran directions test from highway 280 to Cedar Avenue in Northfield, and ended up with the following map:

You can see my difficulty (I hope).  The blue line should be drawn from the start (the top black blob) to the end (the bottom black blob).  (Ignore the blob graphics; this appears to be IE's continued hate-hate relationship with PNG images).  Instead, the path starts to the left of where it should and ends up far short of where it should, having taken an odd detour through some corn fields. If you look at the actual path it should have drawn, you'll see that it's basically correct, but that the path is drawn at a different scale than the map image.  And if you pop up a screen magnifier with pixel coordinates on it, you can calculate that the path on the map image is exactly 125% of the size of the drawn path. Voila! It's the Google Maps bug that's been dogging me all along.

Just to test this into the ground and then some, I reset my DPI to 75% (instead of 125%), or 72 DPI (instead of 120 DPI).  Again, the path was drawn incorrectly, but this time it overshoots the mapping area and the map path is 75% of the drawn path.

It's interesting (to me, at least), that in all of my tests the path appeared to be correctly placed in the top-left-most position.  So, for example, in the image above the first turn taken to the South (as my grandfather would say, the first South turn if you're coming from the East) is actually in the correct place.  I don't know enough (well, okay: anything) about the Google Maps rendering algorithms, so I don't know why this would be.  But after that one correct spot, it all falls apart and you find yourself off in the weeds.  Or the cornfields. Or the bay.

But what if we don't have the problem on our system?

Granted, not everyone has this problem.  You may have never noticed it before I pointed it out, even.  You would need a combination of a high-density display and a non-default DPI setting (which was the case on my laptop out of the box).  But with the increased availability of high-resolution displays, both for desktop and laptops, the problem will only get worse.  It is expected that resolutions of 200 DPI and up will be commonplace in the next few years.  DPI settings would reflect this change and more people will suffer.

What do we do about it?

Frankly, I'm not sure what we do about it; if the browser is telling us it's doing one thing (sizing things according to pixel dimensions) and then doing another (sizing them according to a scale factor applied to the pixel dimensions), where do we go from there?

In a desktop application, this would not be a hard problem to solve; there are APIs to do everything from querying the DPI settings to checking the size of the image on the physical screen.  But in a browser application, there is no such API (that I'm aware of).  We are reduced to try to divine the information from side effects in the environment.  Like trying to predict the weather by how your knee joint feels. Or trying to see whether a plant is poisonous by eating it.

But it's not clear to me now how to do this reverse-engineering; everything I've seen from IE7 makes it seem like my images are the size I think they are; it is only in measuring the screen pixels that I see the error of their ways.

For example, Dmitri pointed me to this interesting blog on the topic.  One of the comments claims that you can determine the DPI settings by creating a div of a particular size and then asking JavaScript how big it is.  So I tried this, creating a div:

and calculating the width/height of the div in JavaScript like this:

    function detectDPI() {
        var IE = (navigator.appName.indexOf("Explorer") != -1);
        if (IE) {
            dpiDetectorElement = document.all.dpiDetector;
        } else {
            dpiDetectorElement = document.getElementById('dpiDetector');
        }
        offsetW = dpiDetectorElement.offsetWidth;
        offsetH = dpiDetectorElement.offsetHeight;
    }

The result?  IE7 reported the offsetW/offsetH as 96x96, and Firefox reported them as 120x120.

Strike out.

Got any other thoughts on finding out this info?  Post it here; I'll try it out.  I'm thinking that there are no obvious, good solutions here yet, or else we wouldn't be seeing the problems in such a widely used application as Google Maps; it's just that people have not yet hit the problem widely.  But they will...

One solution, I feel compelled to point out, would be to use a toolkit more suited to the purpose.  For example, an applet, even if sized incorrectly by the browser, could adjust its size according to the physical size of the image.  In fact, you could even use an applet just to do the DPI detection step (have it detect the size the image it is, the size the browser thinks it is, and then have it adjust the image width/height appropriately).

Let's Get Serious

Posted by chet on September 18, 2006 at 07:27 AM | Permalink | Comments (5)

Some intrepid readers have noticed both the paucity (I love that word!) of posting and the humor-driven content in this blog of late. That is, I haven't posted a lot lately, and what's there has been more in the realm of Java jokes than Java graphics.

Like any good politician, I won't apologize, but I can attempt to explain...

Java and Graphics: Don't worry, I'm still working on technical stuff, some of it very cool indeed. And I'm definitely interested in writing about it. I just don't have anything ready to post yet. There should be some good material coming out when it's good and ready. And some passable stuff coming out before it's completely baked. I promise.

Humor: Sorry, it's just been where my head's at lately. And in my defense, I'll just note that the precedent and tendency was there from the very beginning.
I can't promise not to post jokes here in the future (nor can I promise they'll all be funny). But in an attempt to provide an outlet for general (non-Java-graphics-related) humor, I've created another blog. Like ant poison, it may not actually kill the colony, but at least it may draw the critters away from your Fruit Loops. So if you don't like the jokes, you know a blog to avoid. But if you're brave and foolhardy, check out http://chetchat.blogspot.com. It's a lonely little blog and it would love some company.