Discussion:
FXML and high dpi screens
Danno Ferrin
12 years ago
Permalink
Executive summary: Is there any way in FXML to specify the sizes of the
components in units other than pixels? Either now or in the future?

So I am running a Java Swing Application on a Surface tablet with multiple
embedded FXPanels (mostly to deal with cross toolkit dialog modality... an
issue for another thread). In order to not get the horrible blurred view
(since surface does 150% font scaling) I turn off the dpi fixing for the
java app, so the swing app is pixel to pixel for the most part, with some
dramatic sizing issues with the XP theming (an issue I don't expect ever to
be fixed).

So I have an FXML panel that is supposed to be 560x400, and I literally get
560px by 400px. But the widgets are all sized in EM, which respects the
native DPI scale. I don't want to turn that off, because in this case I
like it. But the widgets scale up, so it sizes like a 373.333x266.666
panel. What I do want to be able to do is specify the FXML sizes in EM, so
I would call it 35em x 25em. Is there a way in FXML to specify these? Or
will I have to do multiple FXMLs mechanically scaling the sizes up. Or is
there some escape sequence or auxiliary data I can add to say "scale pixel
sizes by 150%"?
Milan Kubec
12 years ago
Permalink
Hello,
in general FXML allows the same functionality as regular Java APIs of
JavaFX, where px is used as main unit. So your requirement should be
addressed to JavaFX APIs not to FXML. EM can used in stylesheets, but
that would probably not help you much.

Milan
...
Danno Ferrin
12 years ago
Permalink
I think it should be addressed in the fxml. In fact, I think it can all be
done in com.sun.javafx.fxml.BeanAdapter, at least relative to the default
font, inside the coerce method. The coerce method could be adapted to add
some context object so that the font of the current or enclosing font could
be used too, but for my needs the default font would work

I'll try and get a patch together in the next few days to demonstrate my
thinking.
...
John Smith
12 years ago
Permalink
How about using an expression binding.

For 35em x 25em you could write:

prefWidth="${35*u.em}" prefHeight="${25*u.em}"

It's not 100% concise, but perhaps passable.

These kind of sizing expressions work in scene builder 1.1, which is nice.

-------

Here is an example using a Rectangle to store the width and height modifiers for the fxml file.

<?xml version="1.0" encoding="UTF-8"?>

<?import javafx.scene.control.*?>
<?import javafx.scene.layout.*?>
<?import javafx.scene.shape.*?>

<StackPane xmlns:fx="http://javafx.com/fxml">
<fx:define>
<Rectangle fx:id="s" width="13" height="13"/>
</fx:define>
<AnchorPane id="AnchorPane" maxHeight="-Infinity" maxWidth="-Infinity" minHeight="-Infinity" minWidth="-Infinity" prefHeight="${22 * s.width}" prefWidth="${14 * s.height}">
<children>
<Button layoutX="${4 * s.width}" layoutY="${ 5 * s.height}" prefWidth="${6 * s.width}" text="Top" />
<Button layoutX="${4 * s.width}" layoutY="${10 * s.height}" prefWidth="${6 * s.width}" text="Middle" />
<Button layoutX="${4 * s.width}" layoutY="${15 * s.height}" prefWidth="${6 * s.width}" text="Bottom" />
</children>
</AnchorPane>
</StackPane>

-------

Or instead, you can create your own unit class and use it in your sizing expressions, for example:

package org.jewelsea.measure;

public class Measurement {
private double em;
public void setEm(double em) { this.em = em; }
public double getEm() { return em; }
}

. . .

<?xml version="1.0" encoding="UTF-8"?>

<?import javafx.scene.control.*?>
<?import javafx.scene.layout.*?>
<?import org.jewelsea.measure.Measurement?>
<?scenebuilder-classpath-element .?>

<StackPane xmlns:fx="http://javafx.com/fxml">
<fx:define>
<Measurement fx:id="u" em="26.0" />
</fx:define>
<AnchorPane id="AnchorPane" maxHeight="-Infinity" maxWidth="-Infinity" minHeight="-Infinity" minWidth="-Infinity" prefHeight="${22*u.em}" prefWidth="${14*u.em}">
<children>
<Button layoutX="${4*u.em}" layoutY="${ 5*u.em}" prefWidth="${6*u.em}" text="Top" />
<Button layoutX="${4*u.em}" layoutY="${10*u.em}" prefWidth="${6*u.em}" text="Middle" />
<Button layoutX="${4*u.em}" layoutY="${15*u.em}" prefWidth="${6*u.em}" text="Bottom" />
</children>
</AnchorPane>
</StackPane>
...
Jim Graham
12 years ago
Permalink
The way we deal with platform "stretching" for retina displays is to
tell the platform that we are HiDPI aware and do the "pixel stretching"
ourselves, but this is using only the MacOS APIs for DPI communication.

We have an outstanding Jira issue to do the same thing for Windows 8,
but we haven't investigated what that involves yet, primarily because
there hadn't been a major hardware shipment of Win8 HiDPI screens at the
time we implemented Retina support. It sounds like the new Surface
hardware might be opening that box on the Windows side now.

If we'd implemented retina support entirely by suggesting a different EM
to the CSS code then only code that used CSS for sizing would look good.
The solution we used for Retina scales all content appropriately
including, implicitly, the CSS EM sizing...

...jim
...
Danno Ferrin
12 years ago
Permalink
I'm not to familiar with the nitty gritty of the details of the apps, but
as far as I can tell windows will still respect pixel for pixel sizes, it
will just tell you that the 10pt fonts are 12, 15, or 18px for the default
size.

So like any good issue this has opened multiple venues for a good fix. The
pixel level mashing similar to what is done for apple and retina os
somthing outside of my comfort zone.

However, letting the users specify units in their FXML looks to me to have
utility outside of this issue. For example you could specify your progress
bar as a real percent rather than a float (progress="80%" vs
progress=".8"). I've opened a bug and submitted an inital patch for this
approach. https://javafx-jira.kenai.com/browse/RT-30471
...
Jack Moxley
12 years ago
Permalink
You have to be careful, apples support for pixel densities and screen sizes is very lack luster, it came late in the day and they only ever needed to support a small number of densities and screen sizes.

Googles approach on android, or preferably the standard html5 approach using media queries is going to cause less issues in the long run. The requirement to have the the layout change because of orientation, screen size or density is prevalent in every mobile application I have ever had to write or port.

As such having it done for me, does not appeal, unless I have a way of overriding it.

Sent from my iPhone
...
Richard Bair
12 years ago
Permalink
Hi Jack,

Our experience is that the Apple solution is much cleaner, however you're experience is very interesting and (since you are actually shipping apps on it instead of living in the land of theory) relevant. Do you have a proposal?

One thing to consider is that any scale factor that doesn't end up pixel aligned will result in fuzzy UI. Our "snap to pixel" layout snaps to whole numbers for positions -- but based on the scale factor this may or may not end up pixel aligned (otherwise when animating a UI control it would snap from pixel to pixel instead of smoothly animating through them).

Can you give some examples of the type of control you need, and why?

Thanks!
Richard
...
John Moxley
12 years ago
Permalink
To be fair its been about a year since I have shipped any apps, I am currently working on treasury systems in a bank, so not quite as relevant. :-)

When porting for very small devices, the main rule of thumb was to allow the screen to adapt to larger screens programatically. Lets hypothetically say I was designing an application that displayed a graph of, for arguments sake, sales figures. On a small device I would want a single graph to be displayed, with maybe a button to show the legend in a pop up. As the screen got larger I would want that legend to appear on the graph itself, as it got larger I might want to add options or even more graphs. I certainly wouldn't want the line on the graph to be pixel stretched. I also wouldn't want a massive graph with a button for the legend.

With high resolution screens, apples solution was to basically add 2X to the name of all its images designed for the retina display, this works if all you are doing is doubling your res, but in the android world pixel density is more in flux, and the costs associated with generating so many images is going to be high. Being able to dynamically determine when and where you apply such tactics is invaluable.

Can I suggest reading up on best practices at http://mobiforge.com/ It arose from the .mobi domain back in the day when we were dealing with a mix of wap and html4, it might give an insight into the practice of mobile UI designers.

I think my main point is it has to be adaptable. Android does have too much ui configuration for my liking, but it is powerful. Javafx has got a bigger problem them iOS or Android in that its got to support even more resolutions and devices.

I don't have a proposal, but I will think on it.
...
Daniel Zwolenski
12 years ago
Permalink
Is it worth also looking at web land for strategies here? HTML seems to cater for all screen sizes and resolutions from little androids to massive iMacs.

The Bootstrap strategy is to provide alternate CSS rules for different media types and resolutions.

http://www.onextrapixel.com/2012/11/12/how-to-use-twitter-bootstrap-to-create-a-responsive-website-design/

Just something to throw in the mix.
...
Danno Ferrin
12 years ago
Permalink
The JavaFX CSS as it stands in JDK 7 does not support sizing and
positioning in the CSS (there may have been some work in 8, can't remember).

But since N is small (1.0, 1.25, 1.5, and 2.0) multiple FXMLs may be the
way we solve it, possibly with some mechanical translation.
...
Daniel Zwolenski
12 years ago
Permalink
Agree that this would have to be done currently at the FXML or code level currently, it's not just size, it's also layouts and visibility, alignment, etc.

The 'responsive' web stuff usually divides it up by a few key screen resolutions. It doesn't just change sizes but actual layouts as well.

So anything over 1200 wide uses a fixed width layout with absolute coordinates, centered with white space or a pattern outside of the centered view.

Anything between 900 and 1200 uses a smaller fixed width (ie 900). Resizing the screen from 1201 to 1199 causes it to jump from the larger fixed width to the smaller.

Anything between 900 and iPhone size uses a fluid layout (ie the ui shrinks and grows to fill the space - more like a jfx layout manager approach). We also start to see menus get hidden at this size and things might layout differently.

Anything below that changes to a compact view where a menu bar suddenly becomes a compressed drop down menu, a search box changes into a search button that goes to a separate search page, and horizontal rows get stacked vertically, etc.

It's all totally customizable using media selector rules but those 3 or 4 cut off points seem to cover the 90% useful cases.
...
Tom Eugelink
12 years ago
Permalink
Aren't we mixing up things?

Responsive webdesign is primarily focused at how to show the same information on less screen space, using the same elements in the same resolution. The questions here are: what to hide or add, what to place where.

High DPI screens foremost involves keeping things the same visual size for the user, the actual layout does not change as with responsive webdesign, but the attempt is to have it look the same on all DPIs. The questions here are: how do I keep things readable for the user, how do I keep things sharp (because scaling causes blurry borders, especially when scaling up).

Most approaches at least use a non resolution dependent unit; being it dips or old fashioned inch or cm. If you state that you want the font do be 0.4cm, the device (knowing its dpi) can easily calculate the number of pixels. The blurryness and visual misses a.o. come from rounding; 0.4cm may not be exactly 2x the pixels from 0.8cm. The biggest pains are the graphical elements, because they have to scaled and often get all kinds of minor artifacts that make the whole seem off. Vectors initially seem like a good approach to catch this, but as Kirill has shown (http://www.pushing-pixels.org/2011/11/04/about-those-vector-icons.html), this only works for a limited range of resolution.

So maybe the best approach is a combination of vector and what Android does:
- use a device independent unit and scale accordingly
- optionally split DPI into ranges and provide graphical elements for each range, preferably vector but you can't forbid bitmap, preferably scaling down.

Optionally because not all graphical element need several DPI related versions; a vector circle can be be scaled all the way.

And after solving this DPI thing, then there is the question of available screen size. So you can have (http://en.wikipedia.org/wiki/List_of_displays_by_pixel_density):
- a 3 inch, 200 DPI screen
- a 6 inch, 100 DPI screen

Both have 600 pixels horizontally, but the user only has half (or twice) the visual screen. This means responsive layout. This is something that is solved in code/FXML (layout) and CSS (hide/show).

To keep things understandable I would try to keep those two really separate. So two topics:
- DPI related
- visual sized based layout
...
Richard Bair
12 years ago
Permalink
...
Well put.
...
Note that scene graph units are already device independent pixels. We use the definition of pixel as defined in the CSS spec: http://www.w3.org/TR/CSS2/syndata.html#length-units. This is why we can get away with pixel-doubling everything on retina -- the scene graph says that a node is still 150 width, but that is actually 300 device pixels on a retina display.

Because we auto-scale based on the DPI (although currently *only* for retina) the only thing the application actually has to worry about is that the right images get loaded. At present we use the mac @2x file name trick for images. We could do the same and have @1.5x or @3x or whatever if we had other sizes to support.

As long as you have a way to turn off our own built-in scaling mechanism (there is a command line flag for it, though I'm not sure it is meant as public API yet) you can always put a scale on the root node of the scene and load your own images and you should be good to go.

Richard
Danno Ferrin
12 years ago
Permalink
...
Ok, in JFX 2x how do I get the pixel scale? This commit
http://hg.openjdk.java.net/openjfx/2u/dev/rt/rev/51a85b7f59c2 has it for
Images, but where can I tease it out for the whole display? Private APIs
for 2.2 are OK, but will this be a "wait for 8" feature?
Jim Graham
12 years ago
Permalink
Post by Danno Ferrin
Ok, in JFX 2x how do I get the pixel scale? This commit
http://hg.openjdk.java.net/openjfx/2u/dev/rt/rev/51a85b7f59c2 has it for
Images, but where can I tease it out for the whole display? Private APIs
for 2.2 are OK, but will this be a "wait for 8" feature?
That commit has no public API for anything. The only API it was
creating was for internal purposes.

We have yet to create public API. It will definitely be for 8 only,
though. 2.2 (part of JDK7) will continue to be "do as best we can on
behalf of the app" since we shouldn't be adding new API in a maintenance
release.

The APIs for communicating or customizing the pixel scaling for 8.0 have
not yet been created...

...jim

Jim Graham
12 years ago
Permalink
Post by Jack Moxley
As such having it done for me, does not appeal, unless I have a way of overriding it.
Absolutely we should strive to allow any app to manage its own pixel
registration. But, we do need to do something by default to avoid
completely unreadable apps for those who aren't paying attention to the
screen resolution.

Also note that apps already experience a default high resolution
mechanism in printing. You wouldn't design a printing API that requires
an app developer to draw everything 4-8x larger in order to be able to
read their printouts. But, you do want them to be able to customize
what they print if they want the best quality.

Right now Apple "handles it for you" with pixel stretching. Until we
added retina support we were being managed that way and it got passed on
to our apps as blurriness. We currently "handle it for you" with a
hidden coordinate transform that keeps most things generally crisp, but
we don't yet communicate that default scale to the apps so that they can
customize it further.

For Win8 and MS Surface I think we will probably need to have a default
scale to keep apps from being too small to manage, but it will also be a
little more critical to pass on the info because the non-integer scales
are less friendly to the assumptions that they might make...

...jim
Herve Girod
12 years ago
Permalink
*> You have to be careful, apples support for pixel densities and
screen **> sizes is very lack luster, it came late in the day and they
only ever **> needed to support a small number of densities and screen
sizes.
**> Googles approach on android, or preferably the standard html5 *
*> **approach using media queries is going to cause less issues in the
**> long run. The requirement to have the the layout change because of
**> orientation, screen size or density is prevalent in every mobile
**> application I have ever had to write or port.
**> As such having it done for me, does not appeal, unless I have a way
**> of overriding it.*


I personally find the way Android does this overly complex.

And it's much simpler for them, because they don't use a scene graph,
but only the old way of defining widgets and containers.

AFAIK, their framework does not help at all when using absolute
positioning (AbsoluteLayout on Android), and in that case the
developer must do everything by himself, which is really a pain.

But if they can have this limitation because they only have regular
widgets, it's not possible when using a scene graph and mixing graphic
shapes with widgets like in JavaFX.

Herve
Daniel Zwolenski
12 years ago
Permalink
Some additional comments on this from my go-to guy for mobile development, who does a lot in cross platform mobile dev.
...
Loading...