We ♥ web applications!
At mobalean we love to build innovative web services for Japan and the world. Our experience will help transform your ideas into successful online services.
Mobalean is lead by Henri Servomaa, the original founder and mobile developer. At Mobalean we strive to develop services which are loved by our clients and users. By working in an agile manner, quickly adapting to changing requirements, we can deliver quickly and often.
Hailing from Finland, Henri has a long history with computers and the internet. With a background in Electrical Engineering and Computer Science, he has worked in Japan as Software Developer and System Admin since 2001. In 2005, he joined a company to develop mobile sites for the Japanese market and has been involved in mobile ever since.
Cleve Lendon is a Canadian engineer who has been contracting for Mobalean. He came to Tokyo in 1994, and has lived here ever since. He has broad experience as a software developer, which includes development of mainframe software, Internet applications and mobile apps (Android and iOS). He is especially skilled at writing Java applications (vd. Simredo 4, Grafikilo 15). When not programming, Cleve enjoys improv acting and studying languages, such as Latin and Esperanto.
Our strength is crafting web services for both Japanese and international markets. We bring our technical and cultural experience to help you adapt your ideas into successful products.
We develop with Ruby on Rails and use the best agile practices and tools, such as test driven development and continuous integration to achieve quality.
We are the leading provider of technical expertise about the Japanese mobile web. Mobalean started when the smartphones were just appearing on the market. Our Keitai Web Technology Guide is a quick starting point for learning about the initial challenges of Japanese mobile development. Although the technology stacks have changed since the proliferation of iOS and Android, some of the idiosyncrasies remain. Most notably, the Japanese market is still very much dominated by the big three carriers: DoCoMo, au and Softbank. Developers can find more technical details in our Keitai-Dev Wiki.
Please contact us with your specific requirements.
Email address: email@example.com
If you prefer to call us, feel free to do so under +81 (0)70-6251-7245
For users of Skype, please call mobalean
We've been quite happy developing web sites in Clojure. We use the Leiningen toolchain: test locally, compile an uberjar and deploy it to a Java-provisioned server. But recently we started noticing very slow startup times. By slow, I mean up to 5 minutes before the web application would start responding. Moreover, the problem only appeared on production. Cold starting the same jar locally usually responded after about 10 seconds.
Since uberjars are essentially Java, we needed to generate a thread dump to understand what was going on. There's a good utility, called
jstack which can examine a running process. In our case the dump for a started server process looked like this
Thread dump (kill -QUIT pid): Full thread dump OpenJDK 64-Bit Server VM (24.85-b03 mixed mode): "worker-2" daemon prio=10 tid=0x00007f8ea0007800 nid=0x5b88 in Object.wait() [0x00007f8ed233f000] java.lang.Thread.State: RUNNABLE at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:214) - locked <0x00000000f4a5b440> (a sun.security.provider.SecureRandom) at java.security.SecureRandom.nextBytes(SecureRandom.java:466) - locked <0x00000000f4a5b758> (a java.security.SecureRandom) at crypto.random$bytes.invoke(random.clj:12) at crypto.random$base64.invoke(random.clj:18) at ring.middleware.anti_forgery$session_token.invoke(anti_forgery.clj:13) at ring.middleware.anti_forgery$wrap_anti_forgery$fn__2878.invoke(anti_forgery.clj:67)
The culprit is right there at the top:
SecureRandom, the random number generation. However, we were so intent on finding fault in our application, that we initially skipped over it.
Eventually, after trying various code hacks and looking at the umpteenth trace stopping at
SecureRandom, the light-bulbs came on 💡💡!
Now that we had a lead, it was easy to find this thread on stackoverflow. Essentially what was happening was that our server was running out of entropy. What is entropy? In this case it's random 'environmental' noise, generated by hardware events, such as mouse movements and key presses. These random bits of noise are stored in an 'entropy pool', which is used by
/dev/random to generate random numbers. On headless servers (servers without local users) the entropy pool can dry up, causing
/dev/random to block.
We confirmed this by running
~$ cat /proc/sys/kernel/random/entropy_avail 88which looked rather low, especially compared to my workstation showing
The reason for low entropy has to do with how the pool is maintained in Linux. For more about that (fascinating) subject, see the man page. The Stackoverflow thread has several suggestions for mitigating the problem. Eventually we opted for Havege daemon, since it's readily packaged for Debian.
HAVEGE stands for HArdware Volatile Entropy Gathering and Expansion (homepage).
Here's the description from the Debian package
Debian package: Description-en: Linux entropy source using the HAVEGE algorithm haveged is a userspace entropy daemon which is not dependent upon the standard mechanisms for harvesting randomness for the system entropy pool. This is important in systems with high entropy needs or limited user interaction (e.g. headless servers). haveged uses HAVEGE (HArdware Volatile Entropy Gathering and Expansion) to maintain a 1M pool of random bytes used to fill /dev/random whenever the supply of random bits in dev/random falls below the low water mark of the device.
This is a completely stand-alone daemon which works out of the box. It's memory footprint is also low by today's standards. On a Debian 8, 64bit server
pmap reports a total of 12052K.
After running haveged for a few minutes
~$ cat /proc/sys/kernel/random/entropy_avail 1801That looks healthier!
Now, as expected, the Clojure web application startup was reduced to a few seconds. Yes!!
If you have any comments or suggestions, please tweet us.