SummaDigita blog

This blog has been languishing lately, and I don’t foresee much improvement soon. So just a few quick points to say what’s going on:

• Crescendo has been shelved for now. I actually did some more work on it since the last time I talked about it. I created a few tools to import same data in bulk, namely Go-records with their associated meta-data. That looked very nice, and it was a concrete example of how it could be used by a specialized community. In this case, the Go community that wanted to share game records. I also did the same for movies I screenscraped from the site of Btjunkie. That looked even nicer, because it also showed the movie poster for each entry. The problem with all these examples is the uncertainty of the copyright of all content. That combined with the fact that I just didn’t find a user community eager to use Crescendo led to halting development when the Go example worked satisfactory. It is in a state that can be picked up easily again, but for now the project is dormant.
• The server in my basement crashed and I moved everything to a server in the CloudSigma cloud. Luckily I had everything backed up using CrashPlan (online storage). However, Windows servers are more expensive than Linux servers, so I chose a Debian server to host all the webapps (WordPress, Bugzilla, Tracks). Backup I moved to my mediacenter. The only thing I couldn’t migrate was the Crescendo service because it is .NET-based. However, I explained in the previous point that that is not a big deal. If I pick it up again I’m going to try to compile the source with Mono, which should be possible for the server-side code.
• I’m currently working on another project: creating closed communities for professionals. The first site is up and running at http://www.accountantsforum.nl. It is a community for accountants. The forum is invite-only and only members can invite other members. The basic forum-software is given by Vanilla. I added the possibility to invite contacts from your LinkedIn-network to encourage members to invite others. I have lots of other ideas to add features that are targeted at accountants specifically, it just takes time to do them. Most of these features are useful for other professionals too, so the next site should be easier to set up.

  It is a lot of fun to build extensions on the forum because you have something useful really quickly. It also teaches me new skills, since I was not familiar with PHP and MVC frameworks for web-apps before. Using the LinkedIn API was also interesting. I looked at the Google Maps API yesterday and I’m itching to try that because it looks so easy to create something cool!

• Part of the reason I began working three instead of four days last year was that I felt stuck in my current job at ORTEC. I didn’t feel much progress anymore and I felt I kept doing the same things over and over again. That was hard, because in other respects I was very satisfied with my job. Still I began looking around for other opportunities, and even applied for a job as Team Leader at a company in Bodegraven, where I live. Those talks went great, and I got offered the position…but only if I came to work full-time (5 days a week). Somewhat understandable for a team-leader position, but I had to decline. Fortunately, at Ortec an opening came up for a software architect at the consulting group. They do project work and needed someone to help making the right technical choices. It seemed like a perfect fit and I will soon start there when I have tied up all loose ends in my current position.
• Last but not least, Meike (my wife) was pregnant which required me to run a little harder at home. In the night of last sunday to monday, at 00:05 my son Pepijn was born. He is perfectly healthy and Meike is doing fine as well.

It just doesn’t get any easier than this: Dropdo provides one big button that lets you pick a file on your file system. After uploading you have a URL that you can send to your friends. If they visit that URL, they see a preview in their browser of the file and a big Download button to get a local copy. Also, no advertisements!

I’ve been thinking a lot about how I want to proceed with Crescendo. I have had some misgivings lately about how it turned out so far:

• Firstly, the installation is too cumbersome if not all prerequisites are already installed. It is quite confusing to have to install all kinds of stuff you don’t know what the hell it means when you thought you were installing a little file-sharing application. Or worse, if you click the ClickOnce link and nothing happens.
• Secondly, there is no clear focus on the audience of the application. Is it grandma that wants to share photos of her grandchildren? Or is it the seasoned torrent-sharer that want to supply richer descriptions for the files he/she shares? Is it a social application or are users basically anonymous to each other?

To solve the technical problems with deployment, there are several options. I could keep a native client, but provide a traditional installer. Or I could switch to Silverlight or even an AJAX web application. I debated all these options in my mind, but ultimately I decided that I should not switch to another technology I don’t have much experience with. Of course it would be fun to try something new and I’m all for that, but right now I should be focused on getting a viable product as quickly as possible and that means sticking to the things you know. I don’t intend to change much about the ClickOnce deployment, but I will reduce all prerequisites to just the dependence on .NET 4.0. I should also change the web page to be more helpful. I toyed with the thought of making a ‘Crescendo-lite’: a simple web-app that you could use to quickly submit data to a file host and then send the link to others by email. Then I saw the website of WizDrop, it is more or less the same idea, but perfectly executed.

The second misgiving is about my audience, which was not clearly defined. I more or less assumed I could force a better way of sharing files (by adding meta data) on the world by just building a great GUI. That was kind of naive. Building a GUI is hard, much harder than I anticipated. Of course, a great GUI gives so little friction that it makes all interactions seem simple and obvious, which ironically makes you think it shouldn’t be hard to make. The other reason it was naive is because there are not thousands of people patiently waiting until they can finally use my application. You need to have a clearly-defined group of people you want to reach and you have to build your product around that vision. If you design for everyone then you reach no one.

The concept of entering meta-data when you share something is useful only if you are serious about file-sharing. If you just want to let some other people show your home-made movie, you don’t care about title, director or other characteristics. You just put it on YouTube or whatever and share the link. So Crescendo is not for casual file-sharers and as such it also doesn’t make sense yet to build social capabilities (like the ‘groups’ that are in the application currently). It is however absolutely critical to build search into Crescendo.

Of course, in my previous post I said I couldn’t do two things at once and if I continued learning I wouldn’t be able to develop Crescendo anymore. I can not really decide that I will drop doing either one, so I will try to find time elsewhere. For example, by wasting less time watching epsiodes of Criminal Minds and Psych.

I passed the 70-536 exam last Monday. The reason this blog hasn’t had an update since August was that I can only do one thing at a time and I was studying in my free time. Theoretically I could also have studied at work (for 24 hours) but in practice that wasn’t really an option. My employer did pay the exam fee though, so thanks ORTEC!

The 70-536 exam is one of two exams that are required to obtain the title of Microsoft Certified Technology Specialist. It covers the .Net framework base class library, up to 2.0. So it contains lots of stuff on serialization, security, mailing, globalization, etc. No really interesting things yet: no WPF, WCF, Linq, Entity Framework, and so on. The next exam will be a choice between WCF, ADO, ASP, WPF or WinForms. I already know quite a lot about WCF and WPF, so it makes sense to pursue those. However, WPF is not a prerequisite for the certification that comes after MCTS (namely Microsoft Certified Professional Developer). So, WCF it is.

Learning took an enormous amount of time. I really totally misjudged that. The problem is that the official Training Kit book doesn’t go into the level of detail needed to pass the exam. I therefore went over all the skills required and read the MSDN documentation, making notes as I went (you can find those in my previous post). As it turned out, that was going to the other extreme. The middle road is best, but it is unclear which details are important until you’ve actually done the exam.

I did the exam in Zoetermeer, which is close to where I work (Gouda) and live (Bodegraven), so that was perfect. There is a big parking garage next to the exam site with free parking, which was convenient. After checking in I could immediately start. I was brought to a computer that guided me through the exam (exams are individually timed). I was curious about the process, so I took the time to do the practice exam, even though it is not really necessary. The exam itself consisted of 40 questions for which I had something like two and a half hours to do. Time is not a problem at all, because I was basically finished after 40 minutes and I deliberately took the time to ponder each question carefully. I forced myself to explain why the answer was correct and why the other options were not.

During the exam you can mark questions to review them again later. I marked three questions I was insecure about but ultimately left them the way they were. After reviewing all questions again to acertain I had marked the option I had intended, I finished the exam. You then immediately get your score, ranging from 0 to 1000, where 700 or higher (at least 28 questions answered right) passes the exam. Since I had no wrong answers, I got a score of 1000.

After the exam there was the possibility to comment on the exam. I took the opportunity to give feedback on three questions, most importantly the one where you have to provide two ways of getting some information which they phrased as “which options should you use (mark two)”. I think that should be phrased as “could you use”, because it confused me.

After the exam I was exhausted. I tend to function with peak performance during exams,  but afterwards I am completely empty. Fortunately, I could go home a little early that day.

So, what is next? Development on Crescendo was completely stopped the past months, and I’m not sure I should start it again. I think the concept may be to complex still and the choice of a native client may be a mistake (ClickOnce is not what I hoped it would be). So I’m going to think that over a little bit. As I said, I can only do one thing at a time so if I continue studying for the next exam I will probably do nothing else for the next six months. Where does that leave SummaDigita? On the other hand, if I don’t study I won’t get any further to my goal of becoming a full-time .Net architect.

WordPress is telling me I have already written 700 words, which is probably too long. So I’ll stop and continue when I’ve thought about it some more.

Since there are now two transports available in Crescendo (AnyHub and bittorrent), it has become confusing to understand what happens when content is downloaded. To solve this a logo is displayed next to the row of content like this:

The little logo at the start of the row is the AnyHub logo. I designed the torrent logo myself:

As you can no doubt see, it is a torrential rain of bits. The official logo is copyrighted, so I couldn’t use that. It will have to do for now.

I’m currently studying for the MCTS certification and I have to start with the 70-536 exam. To be honest, learning isn’t much fun. I read through the book during the two weeks of my vacation, expecting to be done. And indeed, the concepts you learn are not very difficult. However, when I started doing some practice questions it became apparent that just learning the concepts is not enough. You have to learn the APIs pretty much by heart. Questions revolve around knowing whether a method accepts a certain argument or not.

This is a problem for me because learning things by heart is something I really suck at. So to make it more manageable I’ve started to reread the book again and this time I take notes of everything that is not plain obvious. Of course, “plain obvious” is a very subjective thing. Nevertheless, these notes may be useful to others so I’m sharing them via Google Documents:

• 70-536 notes – Developing applications that use system types and collections
• 70-536 notes – Implementing service processes, threading, and application domains
• 70-536 notes – Embedding configuration, diagnostic, management, and installation features into a .NET Framework application
• 70-536 notes – Implementing serialization and input output functionality in a .NET Framework application
• 70-536 notes – Improving the security of .NET Framework applications by using the .NET Framework security features
• 70-536 notes – Implementing interoperability, reflection, and mailing functionality in a .NET Framework application
• 70-536 notes – Implementing globalization, drawing, and text manipulation functionality in a .NET Framework application

It is done. I implemented the torrent transport for Crescendo so if you’ve got uTorrent installed you use it in Crescendo.

How does this work again? In Crescendo, there exist so-called content types. A content type specifies certain characteristics for specific content. For example, a content type ‘Chess games’ could have an attribute ‘Black player’. When you submit content for a content type, you can enter values for these attributes. This meta data is stored on the Crescendo server, but the file it describes is uploaded elsewhere. The way this happens is governed by the transport used. So if you submit content and you choose the transport ‘AnyHub’, the meta-data would be stored on the Crescendo server but the actual file would be uploaded to the AnyHub file hosting service.

If you submit content and choose ‘Torrent’ as the transport, again the meta-data is stored on the Crescendo service but the upload is handled differently. A torrent file is created for the files you wish to share and this torrent is passed to your uTorrent client (actually, any client would work as long as it accepts the ‘/Directory’ command line switch). It will then start seeding. A hash of the files is stored on the Crescendo server. If a download is initiated in Crescendo (by clicking the button ‘store locally’ when a row of content is selected), the hash is again passed to the torrent client which will start downloading.

Note that there is no need for a torrent tracker: if another user is sharing the same data in the swarm of file sharers, all necessary data can be deduced from the hash. Therefore, at first you will only see the hash in the torrent client, which will change to the full description of the file.

Today I also implemented some GUI improvements. For example, you will now see a nice list of running transfers with a description of the transfer in progress.

By the way, I signed up for Twitter today. Posts of this blog will be tweeted. I also connected my Google Buzz account because, hey, why not. This gave me the following thought: if all these social networks are reposting the submission on one of the networks, don’t we run the risk of creating an infinite loop in the internet? If my blog posts are tweeted and my tweets are Buzzed and my buzzes are summarized on LinkedIn and my LinkedIn updates are posted on my blog…will the internet crash? I’m afraid to try out. That is certainly not what I want to be remembered for.

It turns out that my statement in the previous post that it was too hard to interface with uTorrent was a bit silly. I have become so used to processes that connect with each other through some kind of remoting via a well defined interface that I had forgotten that the problem is actually much simpler. The Crescendo client is on the same machine as the torrent client. Therefore, you can simply invoke the torrent client and pass suitable commandline parameters. The torrent client will simply create another instance and connect to the instance that is already running if necessary and pass the message to it. More importantly, all the hard work is already done by the torrent client itself.

So all I have to do to use torrent as a transport is create a new torrent using the MonoTorrent library, and invoke uTorrent like this:

uTorrent /Directory <path to content> <path to my .torrent file>/content.torrent

It is as simple as that (the “/Directory” argument is key since without it uTorrent tries to download the files instead of seed them). The beauty of it is that I won’t have to make something specifically for uTorrent and thus require all users of Crescendo to use uTorrent. Instead, I’ll make it configurable what the torrent client is and how it should be invoked. The great thing is that this setup also allows users to make their own torrent handlers that can act for example as a proxy for their seed box.

All in all I’m really pleased about how this turned out and happy that I will be able to add torrent as a supported transport. I’m just baffled it took me so long to think of it.

As was described earlier, Crescendo is a content-sharing platform. It is different from file-sharing networks in that it uses them but is not itself in the business of moving bits and bytes around. Other content-sharing platforms one may be familiar with are iTunes and YouTube, although these host their own files. Crescendo instead only stores the meta-data and leverages other file-sharing networks to act as transports. Crescendo is also different from other content-sharing platforms because it is open: users can define their own kinds of content and share whatever they want.

There are many file-sharing networks available, like bittorrent, edonkey, soulseek, etc. Many file hosts like RapidShare and the like also act as file-sharing networks. Crescendo uses AnyHub to host its files. From the AnyHub FAQ:

..we’re dedicated to providing a pleasant service that doesn’t get in your way.

They mean it. AnyHub allows you to upload and download as you please, without logging in, for files of up to 10GB. It does all this with a very minimalistic and clean interface. Even better (and AnyHub is unique in this as far as I know), it allows you to do this without even displaying a download page and forcing you to wait. Uploads and downloads are accessible through its API. All these features make AnyHub ideal for Crescendo. I contacted the creator of AnyHub and plan to integrate AnyHub and Crescendo better. For the user this should be completely transparant.

Another transport I really wanted to make was a torrent transport. Transporting files through bittorrent is less ideal than file hosting because the concept of ‘seeding’ and peer to peer sharing is not really user-friendly (remember, my grandmother should be able to use Crescendo). However, there are large groups of users that are currently using bittorrent to share files and I wanted to give them the best of both worlds. It would be nice if Crescendo supported torrents so that users that like bittorrent can keep doing what they are doing but can add the extra functionality of Crescendo on top. However, today I decided against adding support for torrents.

What I intended to do was interfacing with uTorrent to put creation and control of the torrents in Crescendo but actually moving bits around to uTorrent. It turns out to be almost impossible. The web UI of uTorrent is incomplete and I can not manage to add a torrent through the web client and get it to start seeding (instead it starts downloading). There is now a new uTorrent API but that requires writing a plugin for uTorrent in HTML and javascript. That is just a bridge too far at this point.

Instead, I’ll be focusing on making the UI less painful and adding social capabilities. For example, it should be possible to create a user group, add content to it, and then invite people you know to access pre-made accounts that are already joined in the group.