So what can we expect for lets say the next month of ILWIS. It’s nice that the 3.8 has been released but that doesn’t mean that the work has suddenly stopped. I have been rather busy the last few weeks with a few developments that I would like to share with you.

1. Tables. The internal design of tables in ILWIS is quite silly (imo). Echoes from a very distant past of Ilwis that were never properly redesigned. The basic assumption of the ( internal) structure is that everything is based on columns instead of records.  Now basically there is nothing wrong with that. Most operations you do (in Ilwis) are column based not record based. Yet internally there is a big flip-flop continuously happening between column based and record based handling of things. This causes a big performance hit. Furthermore because all columns are owners of their own data (and not the table), data access is often going through many layers before getting at the ‘real’ data.
   I redesigned the table structure to a unified system. In the Ilwis 3.8 this already used for loading the vector maps which are basically tables when stored. But these tables are very isolated and don’t have any interaction with tables in general. I am extending that now to the rest of the tables but have to be very careful not to disrupt existing functionality.
   So what do I gain as end-user? Well, tables still work the same as they used to do. But… They load 20x times as fast(realy) and have an access time that is 2-3x as fast. So , yes. You do gain something.
2. Openstreemap tile server. For those who don’t know it. Openstreetmap is an Open database  where geographical vector data is stored free for access by the outside world(see http://wiki.openstreetmap.org/wiki/Main_Page). An Openstreetmap tileserver is a renderer of this data onto tiles of many different zoom levels so that others can use it as layers in their application. I need to support this in Ilwis as it is needed for one of sub-projects of Ilwis and is useful in general. Of course this only works if you have a network connection where there is a tile server.
3. Geonetcast data as OpenDap service. This is work I already started late last year (with success) and I need to polish it up somewhat. Geonetcast is a successful toolbox based on Ilwis, which exposes (see also http://52north.org/communities/earth-observation/ of 52N) loads of RS data in a consistent way. There are whishes from another project I am working with to be able to access this data as a service. The problem is that the data stream is rather large, continuously updating ( from 5 minutes to one day) and in the usual zoo of different raster formats. The toolbox translates this to an Ilwis format to have a consistent and accessible format to use the data in analyses. The data stream is so large though that this can’t be done on a continuous basis. It has to be done per request for a specific dataset(client side). This is the work of the toolbox. For this project I had to do about the same but this time server side(but still per request) and then translate the Ilwis data to netcdf as OpenDap is built around netcdf. What you now basically do is sent a WPS command to an ILWIS or 52N WPS server, it will do the necessary processing server side and then send a OpenDap link back to the client. This seems to work ok; I now included some extra subsetting to reduces the workload for the server.
4. Debugging. As usual after a release you encounter some issues that should have been addressed ( import shape, grrr). Ah well, such is life. I will not make a new 3.8.1 or something but simply replace the current download. The about box will probably show 3.8.01 I suppose.
5. Orfeo.  Already for some time I eyed the orfeo toolbox (http://www.orfeo-toolbox.org/otb/) as a possible extension for Ilwis functionality. But during the making of 3.8 I simply didn’t have time for it. I will make an Ilwis shell for it (both on UI level as on command level) and the include it in Ilwis. If all works well Ilwis will be seamlessly be integrated with the Orfeo functionality. Probably this will not be part of the standard Ilwis download but available as separate plug-in from the earth observation community of 52n.
6. And last but not least Ilwis NG. This is continuous process and progress has been made there.

So what is de planning of all this? Well point 1), 3) and 4) are in the last stages. Have still to do some work there but the bulk is done. I plan the 3.8.01 for end of this month or possibly the first week of June. This will probably not include the table stuff (yet) as I need to do some real testing there if I didn’t mess things up, tables are quite important after all. Orfeo will probably start somewhere mid June. I am unsure at the moment how long this will take but my first guess would be about a month.  And Ilwis NG? Hmm well, difficult question. I suppose that somewhere in 2013 there will be a version but if that already has the same functionality as Ilwis 3.8 I am unsure. Anyway NG will be very modular so I can release it piece by piece. We will see, it is a bit premature to speculate too much about that.

Once again 52Ners braved the elements and gathered for a few rounds of ultimate frisbee. Daniel Nüst instructed the players in the rules of the game, demonstrated helpful throwing techniques and clued us in to which award is most desirable – game/tournament winner, fair play award or party award! We look forward to exploring more of that horizon!

Daniel demonstrates proper throwing techniques.

attentive listeners…

Ready?

Action!

The  ESA is looking for innovative European developers with experience in app development for their App Camp – the ESA App Developer Lab.  Twenty developers will be chosen to meet with like-minded people, create mobile applications using satellite data and gain insight into ESA’s work at its ESRIN location in Frascati (near Rome/Italy) from June 18 to 22, 2012.

Registration is open from April 6 – May 6, 2012 at www.app-camp.eu.

Well, the 3.8 is finally finished. Really? Yeah, think so. The perfect software doesn’t exist. There is always room for improvement, there are always new ideas. But at a certain moment you have to draw a line. The question then becomes “is it good enough?”. In January I decided that, apart from some known missing things, I would wrap things up in April. The beginning of April I looked things over and, apart from the WMS client, all things were done and seemed to be in working order (WMS client was done in time). Sure some issues are still there, some inconsistencies in the UI, but as far as I could see nothing serious. Perfect software takes forever so I decided to release.

So, are we done? Of course not. Undoubtedly there will be a 3.8.1 (no, I don’t say when) and a 3.8.2 is also likely. Maintenance is always needed. There are a few things I have on the menu (in random order)

• Polygon editor. Missing from 3.8 . I now have the technology to make a “real” polygon editor and not that half assed thing that was there in 3.3 and later. But this costs some time.
• Sample set editor. Maybe. I am waiting a bit until people start complaining . I didn’t find the time to finish it ( a very buggy version exists) and until now nobody seems to be missing it. Since I am reconsidering Ilwis’ whole classification stuff  anyway (it has to be improved) I might need a full redesign. This may be related too.
• The Orfeo toolbox. I am considering  integrating the Orfea toolbox (http://www.orfeo-toolbox.org/otb/) in Ilwis as a plug-in. This would extend the functionlity of Ilwis with a lot of cool stuff.
• Improving the speed of tables. Tables are internally based on a silly concept (not my design ) that costs a lot of performance. I already made a prototype for a new design (it is used at a few isolated places in Ilwis 3.8) that loads data files 15-20x as fast (really) and gives access to data about 10x as fast. I need this for big data sets. I need it fast.
• And of course QA. That is always there. There are always new bugs to solve.

So enough to do.

And then there is of course Ilwis NG. The next version of Ilwis. This will probably occupy more and more of my time in the rest of the year. A proposal for a general architecture has been created

These are the picture programmers like to stare at, forgetting all the work that is involved. Most people couldn’t care less about all the squares and arrows, but I thought it might be useful to show that Ilwis NG is no pipe dream. For me the goal is to create an Ilwis version that can function as a kind of connection point between hetrogeneous data sources and diverse processing services while facilitating a good UI – Ilwis as a kind of integrator that delivers a consistent and good UI, be it desktop or browser (something that is sorely lacking in many open source software projects). I also want to create a source base that is much more open for outside developers and much more modular.

In a pilot, I already created a base version of Loader and part of de core data structures for Ilwis NG. This doesn’t do much (apart from loading data) but it is a proof of concept that the ideas that are represented in the diagram are workable. It would be nice to have more developers on board as this would speed up the work and improve the design process.

Members of the 52N staff recently completed a two day first aid training course. Benjamin, Conny, Eike and Henning are now certified first aid heplers! Let’s hope we don’t need them in such a function!