Vague Hope

If you are using the latest JDK 1.7 update 45 and you parse a lot of XML using JaxB, then you may suddenly start encountering this error:

Exception in thread “main” javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1]
Message: JAXP00010001: The parser has encountered more than “64000” entity expansions in this document; this is the limit imposed by the JDK.

You might have seen it when using the AWS Java SDK: https://github.com/aws/aws-sdk-java/issues/123
See here for some more info: https://forums.oracle.com/thread/2594170

So what changed from the previous release (update 40)?
After some digging through the OpenJDK source tree, it looks like there has been some notable churn in the Xerces packages, particularly WRT to (XML)SecurityManager. Note: Not to be confused with java.lang.SecurityManager.

* 8426c41b7922 (Wed Jul 03 2003)
* 32a6df99656c (Tue Jul 09 2003)

At first the latter seems to be the culprit as it touches the lines the exception is thrown from, but closer inspection reveals that commit 32a6df99656c is just a load of refactoring and renaming. Commit 8426c41b7922 appears to be responsible.

What has actually changed here is a (XML)SecurityManger is actually being configured by default, where previously the default seems to have been for none.

I can only guess if this was an intentional change, but it seem sensible to have a security manager by default and perhaps this was the supposed to have been configured by default all along. It seems no many people have hit this problem yet. Given this was (albeit unintentionally) previously disabled, it seems ok to disabled the entity expansion limit again as a short-term work around. At least, better to disable the entity expansion limit than try and pin at an older version of the JDK.

Update 2013/11/6: Looks like this “fix” was back-ported to OpenJDK 1.6 at least on RHEL as of “java-1.6.0-openjdk.x86_64 1:1.6.0.0-1.65.1.11.14.el6_4”. See the RHEL change log. This change is also mentioned in this RHEL security advisory. I would check the source, but hg.openjdk.java.net is down at the moment.

Update 2013/11/13: Seems there is an actual bug open for this now, though currently fix-version is set to ‘8’.

PowerMock is a very handy extension to the Mockito mocking framework. It extends Mockito by allowing the mocking of static methods and final classes. Obviously to do this some interesting mangling is going on inside the JVM. Occasionally it is possible to run into some very confusing error messages.

Yesterday my colleague [Candle] and I spent a very vexing half hour over this puzzle before we figured out the answer. So I though I would share it to see how long it takes others to figure it out. ^^

Consider the following class. Do not worry that it basically does nothing. This is just the minimum to show the puzzle.

import java.net.URL;
import com.google.common.cache.CacheLoader;

public class Loader extends CacheLoader<URL, String> {
	@Override
	public String load (URL u) throws Exception {
		u.openConnection();
		return null;
	}
}

And then consider this test class.

import java.net.URL;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;

@RunWith(PowerMockRunner.class)
@PrepareForTest({ URL.class })
public class LoaderTest {
	@Test
	public void test () throws Exception {
		URL url = PowerMockito.mock(URL.class);
		Loader loader = new Loader();
		loader.load(url);
	}
}

When run this test will fail with the following exception. I have edited it slightly to match the cuttings above. It throws in Loader.load() at the call to u.openConnection().

java.lang.AbstractMethodError: java/net/URLStreamHandler.openConnection(Ljava/net/URL;)Ljava/net/URLConnection;
at java.net.URL.openConnection(URL.java:957)
at Loader.load(Loader.java:7)

Now try and figure out what is going on here. Go go go! And for bonus points, suggest a solution that will fix it enough to make the test pass.

EDIT on 2012-09-15:
Solution

Somewhat late, here is the solution: Add the class under test to the @PrepareForTest statement in the test:

@RunWith(PowerMockRunner.class)
@PrepareForTest({ URL.class, Loader.class })
public class LoaderTest {

The problem is that the URL class is being loaded in such a way that the Loader class sees a different class definition for URL compared to the LoaderTest class. The Loader class gets the real URL class instead of the modified one provided by PowerMockito. Adding the class under test to the @PrepareForTest ensures that the Loader class is loaded with the modified class loaded provided by PowerMockito.

Simple, eh? ^^

For London Green Hackathon I hacked together a proof-of-concept Kindlet (Kindle Applet) for monitoring environmental data in real-time that runs on a Kindle 3. Its actually a generic MQTT client that runs on a Kindle, but MQTT is a very popular protocol for transmitting environmental data so I think I can argue the ‘green’ values of this project. After all, Nanode is just an Arduino with an Ethernet port, that gets green badges.

All the code was written after 1AM, thought the night and into the next day. Its somewhat rough around the edges. But given we were not sure if it would work at all, I am very happy with the result.

The Kindle is NOT intended as a general use computing device. There is a SDK, but its closed and so far as I know very little has been done with it. Its basically for creating interactive books, not applications. Seems current bets are that is will remain this way. Hardware limitations aside, the software is really quite fragile. I am guessing there is a single JVM process for everything, and there is no sand-boxing. Any unhandled exceptions trigger a JVM restart.

Installing apps on a Kindle requires several hacks to be applied to the device first. In order: jail-brake, usb-network, developer certificate and changes to the JVM security manager configuration. The last one is not required for all apps, but it was needed for this hack.

Jail-brake and USB network: That page has all the required details and links to the jail-brake downloads.
If the USB net freezes on you, try this.

All Kindlets must be signed. Most people share the same certificate (keeps things simple) and this needs to be copied into `/var/local/java/keystore/developer.keystore’ on Kindle.

Java security manager: MQTT requires permission to open a socket. By default the Kindle only allows Kindlets to do HTTP and HTTPS. By editing `/opt/amazon/ebook/security/external.policy’ and adding `permission java.net.SocketPermission “*:80-“, “accept, connect, listen, resolve”;’ we can get around this blocker.

Writing Kindlets: this page is a very good getting started guild. The Hello World example Kindlet is a useful starting point. I think this page also helped.

AWT / Kindle GUI: JavaDoc for the Kindle GUI tools. Its basically AWT with a custom set of widgets specially for the Kindle. Not the most clear guide but a good start. Generally I was able to deduce the remaining 10% of data by trial-and-try-angai-until-it-works-dam-it.

But the final result? A funky Kindlet that can connect to any MQTT broker and tail the content of a topic space. Oh, and a greatly increased knowledge of how a Kindle hangs together. It really is an awesome device.

Music, films, acting, painting… art of any kind.  People always have an opinion.  It is a completely open-access system.  No special skills are required to understand and enjoy art… what would be the point in music that was only interesting to skilled musicians?

A display of great skill earns great praise and respect, possibly fame.  No one wants to know how a given tune was dreamed up, the tedious months of refining the composition, the 100 failed songs that preceded this one.  The art in a work of art is all in the interpretation, the mind of the observer.

Programming, coding, the process of making software.  Call it what you will.  I will go out on limb here and suggest that, for MEDCs at lest, software is more ubiquitous that art.  Indeed all films, music, games are themselves dependent on software.  As important as paint is to an artist.

But when an artist creates a great painting, where is the credit to the chemist who create the paint?  Paint is commodity.  The beauty in any technique is its easy of repetition.  A paint only has to be created once to benefit many artists.  But the paint is too easy to obtain.  In perfecting a manufacturing process the chemist has, ironically, removed their name from the world.  No fame or praise is afforded to them.

A job done too well is never acknowledged as having ever existed.

What drives an artist?  What do they seek?  Material gain, certainly, but I suspect that is not all.  To want to show off is to be alive.  To have your existence acknowledged and counted.  To prove that you are not just like everyone else.  That there is something that only you can do.  To a musician performing live in front of a large audience there must be such a sense of acknowledgement.

My fate now is writing software.  It has become a reflex – something not requiring exerted effort.  Thought and planning, yes.  But like the musician playing from memory, the code just flows.

I never planned to be a software engineer.  In a way, I tried quite hard not to be.  The default choice, the easy and obvious choice, is usually the wrong one.  Always taking the easy path leads to a dead end.  No, worse than that.  It leads to a straight path of more of the same.

There is no performance in programming.  With a life time of experience and honed skill and craft there is still no grand display, no art.  No fame, no praise.  The great enablers of technology so often unknown, lost in history.

In a world of billions, strive to set yourself apart and assert your existence.  Be noticed, be remembered.  In software, is this possible?  Will more remain than unfixed bugs and sarcastic commit comments?

How many are remembered for their code?  Ideas yes, but the art of coding?  If code is incomprehensible to non-coders, then can such even be possible?

A common theme in the Earthsea books is that it takes power to know power.  You can not comprehend someone’s skill without first possessing a near level of skill yourself.  People have always mistrusted magic.

This is a blog entry that I have been thinking about writing for years.  It has then been hastily written and sat on for several weeks.  Apologies for the D&D reference in the title.