Saturday, August 22

Using MPS to Keep Several SDKs Up-To-Date

It has been very long time since my last post.
This time I'll like to introduce an idea, not a complete solution.
I'm working in a team responsible for keeping 5 SDKs in ship-shape and up-to-date. Each SDK is written in different platform and also different language. To name a few iOS with Obj-C, Android with Java, Unity with C#, Cordova with JavaScript and Adobe AIR with ActionScript3.

Maintain and develop all of those SDKs manually is never-ending work. Even if we'll scale the team and have a team per SDK it will still be hard to maintain because then fragmentation of the different SDKs will happen even more frequently than one team.

After research several directions I've decide to concentrate on using JetBrains Meta Programming System (MPS) to create one meta-language from which we will be able to generate all the different SDKs source code.

It is important to note I'm not trying to generate the entire SDK in each platform, only the parts that are not platform specific which in our case are also the parts that change frequently.

This diagram show the flow

In future posts I'll describe each of those steps:

  1. Building MPS counterpart of each language
  2. Building TextGen for each language
  3. Building X language (I haven't found a name yet)
  4. Building model-to-model transformation for each language
  5. Build scripts to combine each SDK platform-specific part with the auto-generated part
  6. Hidden parts that I'm not able to see yet (please leave a comment if you do)

Tuesday, August 5

ProtoBuf in Unity3D

Recently I needed to add two way communication to Unity3D plugin I worked on. We decide to use plain-old-sockets and use Protocol Buffers to serialize and deserialize the messages.
We already had the proto file that define the messages and the server side is already written so all I needed was ProtoBuf implementation for .NET. The main problem is that most C# libraries for ProtoBuf either use attributes and not proto files or they use reflection which is not supported in Ahead-Of-Time (AOT) environment like Unity3D on iOS.

After some research I decide to use ProtoBuf C# Code Generator project by Peter Hultqvist. The main difference between this and other ProtoBuf implementations is that this one create C# code that does not use reflection and does not depend on binary libraries. It generate 3 files:

  1. The clean messages file that contain only the properties of each message
  2. The somewhat messy serialization/deserialization counterpart of each message.
  3. Helper classes
I use this library on Mac OS X with mono and it run without a problem. It generate clean code that you can easily read.

The few minor issues I did find were quickly (2 hours) fixed by Peter.

Things to note

Api Compatibility Level

One important notice for Unity3D developers: because the current generated code uses InvalidDataException which is not part of .NET 2.0 subset. If you don't to the following change the project will fail when building iOS XCode project with error message that not explain what is wrong.
You must do the following before exporting to iOS XCode project:
  1. Open Build Settings window
  2. Select iOS
  3. Click Player Settings... button
  4. Under Optimization section change Api Compatibility Level to .NET 2.0

Reading Message

If you want to read message from a socket you will need to wrap your Socket instance with NetworkStream and then wrap that with SilentOrbit.ProtocolBuffers.StreamRead. This might be confusing because there is already StreamReader class in .NET, I point that to Peter and I'm sure he will find other name.
If you don't wrap your NetworkStream with StreamRead you'll find that the underlying code uses Position property which is off course not available in NetworkStream.

Using Socket in Unity3D without Pro License

Recently I needed to add to our Unity3D plugin the ability to talk to another application via sockets. I'm a long time .NET developer and sockets is peace of cake to use. The problem start when I test the application on iPad device. During the building of XCode project from Unity I got the exception:

Error building Player: SystemException: System.Net.Sockets are supported only on Unity iOS Pro. Referenced from assembly 'Assembly-CSharp'.

As it turns out Unity3D AOT compiler check and fail if any of your script or the plugins you have uses System.Net.Sockets namespace without having pro license.

The solution is to use a plugin call Good Old Sockets which you can find right in the Assets Store of Unity3D. This library require only to change the namespace you are using from System.Net.Sockets to LostPolygon.System.Net.Sockets. The price is 25$ at this time and their license is very open, you can use and distribute this library without extra cost.

I had very quick response for support from LostPolygon both in mail and in Skype.

Saturday, March 22

Generate PDF with Sphinx Documentation System


One of the most important things in any project is documentation, yet it is usually the last thing if it is done at all.

I've recently change jobs and in the new company we create SDKs for different platform and languages. We are lean and fast and do a lot of changes quickly but we realize we better get our documentation abilities agile as well as our development skills.

What is Sphinx?

After evaluating several documentation tools I've found Sphinx. This is the tool used for Python documentation. The reason it looks best for me is that it is a mature project, it has a notion of version which is important for documentation as it is important for the software itself. It has many extensions that are not required for me now but I'm sure will come in handy later.

Output to HTML

The first and most important requirement is that the tool will be able to output HTML based documentation. This is because we plan to have each specific platform's SDK created in it native tool and use HTML as the one unified languages to connect them all together (thank you hyperlink).
If you follow the quick-start application of sphinx all you need to do to create HTML docs is 

make html

Output to PDF

This is not a must but it will be nice to be able to create PDF documentation both for download and in-browser reading. The problem is that when I run make latexpdf I got this error:

make[1]: pdflatex: No such file or directory

After google it for a bit I was able to find a post about how to generate PDF  using Sphinx. From there I was able to find the line after pdflatex command as follows:
This is pdfTeXk, Version 3.141592-1.40.3 (Web2C 7.5.6)
Using this I was able to find MacTeX-2013 Distribution which is Mac package for TeX tools. The problem is that the main download is 2.2GB download but a few clicks away is Smaller packages for only the essential tools.

Wednesday, December 4

SproutCore Web Application Development - Free eBooks

A new book about SproutCore has been published and I've has the honer to review it. The book name is SproutCore Web Application Development and it explain everything you need to know to get started with SproutCore.
The debate about which MV* JavaScript framework is the best has raising flames. SproutCore does not get the attention is deserves as a proven stable framework that support sites like iCloud (used to be known as MobileMe).
One area in which SproutCore did lack was documentation. This problem has been fixed and this book is one peace of prof.

I'll be publishing posts next week, anyone who will post a comment in those posts will be candidate for FREE copy of SproutCore Web Application Development book

Monday, September 9

1111 MSTestRunner Users

I'm happy to see today that MSTestRunner Jenkins Plugin I wrote and use every day has pass the 1K users this month.

Many thanks to all the users and specially to those who left comments explain how to improve it.

Have fun using it and keep on testing.

Tuesday, August 27

CouchDB Many To Many Joins

I'm on the process of converting our application from using SQL Server and Entity Framework 5 to CouchDB. I'll explain the motivation in separate post.
This post is about entity relationship of Many to Many.

CouchDB does not have entity relationship the way RDBMS does. But it's not hard to create them.
Our application is multi-tenant and each user in our application can be allowed to access many tenants. This makes a many to many relationship between users and tenants.

First thing is modeling users and tenants as documents. I've took the very simple approach of creating a document for each user and each tenant. User document have field call $type with value of "user" and Tenant document have field call $type with value of "tenant". Simple :)
I use dollar sign in the field name to make it very clear it is not a native field of the document itself, it is a "system" field.

The way I've decide to model this relationship is by holding array of User document IDs inside the Tenant document. Generally you will want to hold the array on the side that is least likely to change.

The documents will look something like this:

  • _id"54",
  • _rev"3-53c1c569bc6c316d4d29f20c47f05184",
  • $type"user",
  • username"",
  • firstName"ido",
  • lastName"ran",
  • email"",
  • passwordSalt"AABBCC=",
  • password"123HASH"

  • _id"20121207153325-Ido-Test",
  • _rev"2-8d9cb3b070a062d9a65ed22fa3f7bf9b",
  • $type"tenant",
  • users:
    • "54",
    • "63",
    • "84",
    • "124"
  • name"Ido Test"

Selecting the join of both users and tenants is done in two different views, one for selecting tenant(s) and their related user(s) and the second to select user(s) and their related tenant(s).

In order to select user and the related tenants we use CouchDB "Join" view as explained by Christopher Lenz:

function(doc) {
  if (doc.$type === 'user') {
    emit([doc._id, 0], null);
  else if (doc.$type === 'tenant') {
    for (var i in doc.users) {
      emit([doc.users[i], 1], null);

In order to select tenants and the related users we use feature of CouchDB called Linked Documents. The feature allow us to emit values of structure { _id: "linked-document-id" } which allow us to later query the view with include_docs=true to select in one GET all the documents including the linked documents.

function(doc) {
  if (doc.$type === 'tenant') {
    emit([doc._id, 0], null);
    for (var i in doc.users) {
      emit([doc._id, 1], { _id: doc.users[i] });

In conclusion, it's pretty easy to have many-to-many relation in CouchDB. Storing the relation in one type of document gives some level of integrity but not transactional integrity.
The same views can be used to have One-To-Many relation but that will be in another post.