Software developer blog

Test driven development and unit testing frameworks

In the last few years I've been using test driven development, and grew really accustomed to it. TDD not only helps in avoiding strange errors, it also helps in software design, implementation and enforces a pro-active approach towards debugging. At first it may seem that writing tests for each and every small part of the code is time well wasted, but in fact it is definitely worth the effort, especially if an automated unit test framework is used. As not everyone may be convinced at first, here is a quick summary about how TDD works, how it improves productivity, and what are it's pros and cons.

The work-flow starts by writing the test using an automated unit test framework, and see it fail. There are several reasons for first writing the test, and implementing the feature afterwards, but I'll get to that later. After the test fails as it should, implement the new feature, and test again. If the test still fails, fix the code, and test again. If your new feature works, but tests of earlier features fail, than see how the broken code relies on the functionality you have changed, and make sure that you fix all bugs. Once all test pass, commit your code to the version control system, and continue by writing the next test, for your next feature. It is important, that you always write only as much code, as necessary to pass the test.

@ //

We are crawling at last...

The last few weeks were really busy, and exciting. Until now I could not reveal any of the details of what we were up to, but a few weeks ago the project went public, and on-line: please feel free to visit the Erdős WebGraph project.

About two month ago we were asked to create a graph of domains where the directed edges represent hyper links. Our first attempt was to install several of the publicly available crawlers, but those all had indexing features that we did not need, and neither seemed lightweight enough for our extremely limited hardware capacities. Another inconvenience arouse from the almost complete lack of documentation for these software. Finally we decided to implement our own crawler.

@ //

Postponable iterator

When I started this blog I had no idea how hard it will be to post useful things without revealing too much about the "top secret" projects we are involved in. 🙂 This time though I've been allowed to release an entire class that turned out to be useful in an application we were developing a while back. The piece of software below was created with a financial support by Uratim Ltd. and can be used under the terms of the GPL license. YAY for free software.

Postponable Iterator source

The iterator pattern is one of the most frequently occurring patterns used in software design. Even before one learns about design patterns inevitably implements an iterator without actually realizing. The problem I came across not so long ago was that I had an iterator that was returning  jobs from job packets that would have to be processed but there were a number of constrains I had to keep in mind:

  • For each job there is a resource assigned for the job. It can only be executed using the resource assigned to it.
  • As long as possible certain resources should not be accessed too frequently. (But they can be accessed more frequently when it is necessary.)
  • Packages are processed semi asynchronously: all jobs in one package are returned at once, but there may be more that one pending package. The maximal number of packages under simultaneous processing is a parameter.
  • When there are too many jobs waiting for a single resource than access times should be gradually decreased to avoid too long waiting times for jobs.
@ //