in October, I spent quite a bit of time optimizing this release, shaving off an estimated 60-70% of queries through some smart caching and removed redundancies. In addition, a new database server is now in place. Together, these should get us quite far in the "Fast Server" agenda.

We're however still not done, and now we will do more optimizations that will be part of 8.09.3, due next week, and will move static files (images and javascript) to a separate lighttpd instance to remove that load from the normal Apache. When all this is done, the Maemo community should have infrastructure that will be able to serve it for a long time.

In addition to optimization, we've been working on some other features related to the website:

• Many new activities, such as Internet Tablet talk and Maemo Wiki now count towards karma, with more karma refinements on the way
• Issues with weird email formatting in Maemo forums has been fixed and will be part of 8.09.3
• Per-module karma listings are now showing correctly, making it possible to see the top contributors to different aspects of the community

Midgard is a big and complex piece of software. If you notice any issues related to this upgrade, please let us know. And if you have any ideas on improving the website, be sure to file those too!

Technorati Tags: lighttpd, maemo, midcom, midgard, ragnaroek

The reasons for this are quite simple:

• Ubuntu follows the same synchronized release model as Midgard does, offering a predictable upgrade schedule
• In many cases the stable Ubuntu version has newer PHP and GNOME packages than debian stable. As these are Midgard dependencies this makes our life easier
• Ubuntu is one of the distributions supported by the openSUSE Build Service
• Ubuntu is still basically the same familiar rock solid platform that debian is

While not everybody is happy about the way Ubuntu has established itself in the market, there seems to be a strong gravitation towards switching to it, on both desktop and server.

Thursday

• Sebastian Bergmann gave us a training session on PHP software testing, as as result we now have over 1000 unit tests for the MgdSchema API for PHP. We also looked at Selenium for acceptance testing and deployed a continuous integration server for MidCOM 3 development

Friday

• We gave my Midgard replication talk and Tero's PHP-Python signaling workshop from FSCONS
• Then we discussed new UI and visual guidelines for Midgard 2
• ...and hacked new MidCOM 3 features like component inheritance
• Solt and Piotras started working on getting Midgard to compile on Windows
• In the evening was TKY's 136th anniversary party, and sauna evening at the Midgard Gathering facilities

Saturday

• Saturday started with waking up the various Midgardians sleeping in OK20
• Alexander gave a presentation on how Midgard's internals could be ported from straight C into Vala
• We discussed reorganizing Midgard's documentation and appointed Solt and Neithan to be in charge of it. I expect to see lots of documentation-related bugs in Trac
• Split in MidCOM between "core" and "contrib" components was clarified and we made an updated list of core components and discussed some components we hope to promote to core soon
• Midgard 9.03 feature list is now decided. With 9.03 we will focus on the persistent storage and MVC layers of Midgard, and plan to add the CMS features on the application level around 9.09
• Rambo, in his role as a VCS Tyrant started reorganizing Midgard's SVN
• We started planning a foundation to govern the Midgard project. This will not be a big thing like Mozilla and GNOME foundations, but mostly a practical instrument for handling copyrights and Midgard Gathering arrangements
• In the evening we went to Kaisla for some beers to celebrate the fact that Midgard actually did compile (though not run) on Windows

Some pictures on Flickr.

The Midgard Project switched to a new synchronized release model with the 8.09 "Ragnaroek LTS" release. Synchronized release model means that a major release of Midgard will happen every six months, tuned to be part of the larger Linux software ecosystem as described by Mark Shuttleworth:

WHAT IF you knew that the next long-term supported releases of Ubuntu, Debian, Red Hat and Novell Linux would all have the same major versions of kernel, GCC, X, GNOME, KDE, OO.o and Mozilla. Would that make a major difference for you? I’m willing to bet not - that from a customer view, folks who prefer X will still prefer X. A person who prefers Red Hat will stick with Red Hat. But from a developer view, would that make it easier to collaborate? Dramatically so.

...

The same model has been employed very successfully by major projects like GNOME, Eclipse and the Ubuntu distribution. As Midgard relies on many GNOME technologies, being in the same release cycle with them helps our development immensely, and makes Midgard more easy to install because we have a more stable set of dependencies.

In this case we're running a query and checking if a different user is allowed to see something, as specified by Gadgetopia:

Hey, CMS, I have this list of content IDs here… How did I get them? Yeah, well, that’s not that important right now…

Anyway, can you look at this and tell me which ones I can show Nathaniel Snerpis? Here, just take them all, and give me back the ones I can show him.

With Midgard you do this by fetching the objects first with Query Builder, and then checking their permissions via MidCOM auth service. When user is logged in, his ACLs are already automatically applied to the results so no checks are needed.

<?php
// In this example we have some pre-gathered list of GUIDs
$guids_array
(
    '35c2b080736b11dd935ab300c51623d723d7',
    '18aa22ee736b11dd935ab300c51623d723d7',
);

// Instantiate a query builder
$qb = midcom_db_article::new_query_builder();
$qb->add_constraint('guid', 'IN', $guids_array);

// Get the articles
$articles = $qb->execute();

// We need Nathaniel Snerpis' GUID to perform the ACL check
$nathaniel = $_MIDCOM->auth->get_user_by_name('nathaniel');

// Loop through the articles and check permissions
foreach ($articles as $article)
{
    // Check if Nathaniel is allowed to see this
    if ($_MIDCOM->auth->can_do('midgard:read', $article, $nathaniel))
    {
        // Show the article to Nathaniel
    }
}
?>

You can see here that we're using GUIDs, not IDs to refer to content. This is because GUIDs (compliant with the UUID spec) are replication-safe, and you can trust them to be the same on every system in your Midgard cluster.

Note: in this case the API example was in PHP. But with Midgard 9.03 you will be able to use the exact same APIs with Python and Mono.

Clear vision

One important thing we need to settle on is a clear vision for Midgard. It should be a vision everybody in the community can agree with, and a vision that will benefit the user base.

I believe that "your data, everywhere" is the right idea. We should target Midgard for the replicated world where your application data can be easily transferred between multiple devices, served on the web, and shared with your friends.

While some believe that soon you will "never be offline", I think that level of ubiquitous connectivity is either unattainable or at least very far off in the future. What we need instead is to be able to have our data with us, wherever we go.

Unique visual identity

What really sets Apple's products apart is the "fit and finish," the ultimate impression that results from thousands of tiny decisions that go into a product's development. Take Apple's pioneering work in injection molding. It's part science, part art, and plenty of trial and error. (Who Is Jonathan Ive?)

Design is often what differentiates winners from the mass. Apple's beautiful laptops and cell phones are a good example of how you can beat a crowded market by focusing on design and details.

While we should have distinctive visual elements to build our user interfaces from, I don't mean only that. By design I mean that we really need to think hard of what our users need, and design beautiful interfaces that make things easy for them.

On the visual end, the Midgard colors of grey, orange and brown should stand out quite well. We have excellent icons done in the Tango style, and many other design elements coming from Ilkka Martio's work on Midgard2 styling. These should be cornerstones of the new visual user interface. In '99 Midgard had a strong, distinctive visual identity. We need to rebuild that.

Content management

Content management: This is the everything that happens with a piece of content after its created and until it dies. This includes the permissions, workflows, versioning, check in/out, task management, reporting, archiving, administrative searching, language translation, and any other action involved in keeping this content relevant, current, effective, and general under control. (The Four Disciplines of Content Management)

Midgard's origins have been in web publishing, and through strong templating and component systems this is where we've been quite good at. But content management we haven't done that well. At least, if content management is taken to a wider meaning than just "making stuff editable and published on the web".

With the new Midgard we have the delightful opportunity to rethink this all. Thanks to our active consulting companies and the powerful MVC framework we shouldn't focus on actual components this time - they will appear for sure through various client projects. Instead, we should concentrate on the basic ideas of content management.

This means that we should think of how the content is actually edited and published, and how it lives on in a service. Handling and connecting of content fragments, managing inter-document linkage, and versioning are all important features. Similarly, we should make it easy to send documents to commenting and approval rounds, and to serve them in multiple variants.

An important piece for this would be making the notification service that in older Midgard existed as a separate component an integral part of the system. Any Midgard application should be able to store and show notifications for users. If this is utilized from core up, then workflow between different parties working on a site or a document will be much easier.

To ensure consistency between components, and that components are as easy to develop as possible, MidCOM should provide a good set of controller baseclasses for typical content management activities.

Replication everywhere

Before his retirement as Microsoft's chairman, Bill Gates suggested that a so-called pervasive desktop would be a focus of Windows 7, giving users a way to take all their data, desktop settings, bookmarks, and the like from one computer to another--presumably as long as all those computers were running Windows 7. (PC World)

Even though Midgard has supported replication since 2000, it has always been a bit of an extra feature. Most installations have still been on the server end, with the server acting as an isolated island.

Following our "your data, everywhere" vision, this must change. Replication tools must be shipped with every Midgard installation, be it on server, on desktop, or on a mobile device. And they must be integrated with the user interfaces. Anywhere you edit something, you should be able to share it with others, and see where that piece of content originally came from, and who it has been shared with.

When online, the sharing and synchronization must happen instantly. When offline, the changes to be replicated must be collected into a local spool, and then sent over when connectivity is available.

Separating content objects from internal objects

...load-bearing walls in content management — those things that require a content editor to call a developer to change them. Where do you draw the boundaries? (Load-Bearing Walls in Content Management)

Everything in Midgard is an MgdSchema object, be it permission, site template, user, or an article. This is good in the sense that it allows familiar programming interfaces to be used on every level of the system. But it also means that internal objects sometimes end up being used as content objects, causing confusion.

For instance, persons and groups are often used as on-site content. But they differ from other pieces of content on many levels, for instance because they have so many other objects connected to them through permissions and metadata. Because of this they work with different rules regarding to deletion and approval than others, which many users can find frustrating.

We should see to have stronger differentiation between content objects of different types to ensure smoother usage and replication.

Quality

Quality criterias - Portability: Easy to configure in new environments, Few OS and webserver dependencies, Few and documented version dependencies (language version, database version, external libraries), Able to survive environment updates (Quality in PHP Projects Beyond Unittests)

Like many other pieces of free software, Midgard has been riddled by quality issues. While our bugs haven't meant gaping security holes, changing APIs and memory leaks have caused many grey hairs to our users.

To improve quality in the Midgard software stack we have already taken some steps. The new synchronized release model ensures that Midgard and MidCOM play well together and that component developers can start using new interfaces and features more quickly. It also makes Midgard more predictable to users, helping them to plan their upgrades well ahead.

The openSUSE Build Service has also increased the quality of Midgard software. Now we can provide prebuilt binaries to most popular Linux distributions, making installing and upgrading Midgard easier. The new datagard tool makes setting up a Midgard site a single shell command, which is also an important step here. In the future we should aim also for the virtualized systems by using tools like SUSE Studio.

Quality and backwards compatibility of commits is also important, as are coding standards in a major project like this. To that end, our new VCS Tyrant should be able to help our contributors in their work. Similarly, the newly-revitalized issue tracker with its clear milestones make tracking the development work easier.

Finally, better software testing is a major step to strive for. Regressions have hurt us in the past, and the only way to get rid of them is to to implement a good test suite. For this, we will have an unit testing workshop on the first day of the Gathering.

Documentation

If our intent is to encourage people to use these tools and to contribute their time to the development of these tools, we need to spend more energy on introductory documentation. Engineers who look at Sourceforge are looking either for solutions to problems or are looking for interesting tools. In either case the Web site needs to convey the important details in a short form to enable the reader to gauge their level of interest. Obscurity in documentation benefits no one as it annoys potential users and non-users alike. (Is Documentation Holding Open Source Back?)

Midgard's documentation wiki is currently in a horrible shape. As Midgard's capabilities have grown, and versions changed, new stuff has been added. But all of the old stuff is also there. And users, stumbling there either through the navigation or a google search, will have few ways to tell whether a document is up-to-date.

Developers clearly feel this frustration, and as result much of the newer documentation, especially for Midgard2 is available only in blog format.

Because of this, we really need to put an effort to improving the level of documentation. And important first step would be to take the current wiki content into control. Old information must be either removed or tagged to the version it belongs. Templates must be tuned so that they show clearly if something is current or historical information. And the new materials must be collected from the various blogs in order to be published on the site.

After this is done we must rethink the way the wiki is maintained. Maybe we need a "documentation master" just like we now have a "bug master" and a "VCS tyrant". Maybe we must switch to MediaWiki. Or maybe we must make the wiki a replicated system that we all can run and edit also on our own machines. As I am not quite sure of the right solution this must be discussed well in the Gathering.

Virtual servers

One of the more successful technologies to have been developed is virtualization. Broadly speaking, to virtualize is to make a single piece of hardware function as multiple pieces. Different user interfaces isolate portions of the hardware, and make each one operate as a separate entity. As applied to data centers, installing virtual infrastructure allows more operating systems and applications to run on fewer servers, which reduces overall energy use and cooling requirements. (Ecopreneurist)

In the past Midgard has been hurt much by the fact that root access is required. Less advanced content management systems have surpassed us in popularity by many orders of magnitude because anybody can set them up in just some web space.

But now I believe things are changing. Virtual servers are becoming cheaper and cheaper, and as they allow much larger degree of freedom than shared web hosting, companies and even individuals are now buying them instead. And on a virtual server you can run Midgard easily.

If we want to really benefit from the wide popularity of virtual servers, we must take some steps. First of all, Midgard must be very easy to install, as it is now thanks to datagard and OBS. And secondly, we should provide preconfigured "virtual appliance" images that hosting providers running Xen or VMware can set up just as easily as they would an off-the-shelf Debian or RHEL.

The replicated network setup should also take this into account. Users should be able to easily keep adding new virtual servers into their cluster and have that instantly start sharing data with their other servers.

Their webmaster and Nemein alumnus Heikki asked me for some history of Midgard, and this is what I wrote:

Well, original Midgard was completely programmed in order to run the Grey Wolves site. We hacked it together with @jlz in Helsinki Capital of Culture offices in Etelaranta at night time.

Before Midgard, the site had run on a SGML pipelines -based publishing system we called SusiSGML, and member registry on some custom Perl stuff. Then we wanted more dynamic functionality, and Midgard was born. At first @jlz was intending to write it in Scheme, but then chose the C+PHP extension.

For administration, we wrote the “Midgard Admin” tool, traces of which can still be seen in SpiderAdmin.

Midgard was intended to be a one-off, but then as my employer Stonesoft clearly needed some system I deployed there. Then @jlz was hired to my team, and we decided to open source Midgard before any IPR issues would rise with our employer, which was after all a proprietary software vendor.

When the original Midgard-based Grey Wolves site launched in early 99, we had the great example of what things can be: Hiiden Hirven Hiihto photos and reportage were live only couple of hours after the event ended. And this was before we had digital cameras…

Jukka Zitting, the original Midgard lead developer also added some points:

Architecturally SusiSGML was pretty much like early versions of Apache Cocoon, only with SGML and DSSSL instead of XML and XSL. Had we been at it a few years later, Midgard might have been based on Cocoon. :-)

The Midgard project will turn 10 years old next May, and so it is great time to collect some odd bits and pieces of Midgard folklore.

I immediately added three apps, displaying my latest conference presentations, upcoming trip plans and posts on my blog:

This is great as the applications already were part of my workflow. However, in general it is not a good idea to trust web apps to be the only place where you keep your things.

In my case, I'm using my Midgard-powered website as the backup where I automatically gather information I publish on other services like Jaiku or Flickr. This means that I can use the convenience and collaborative features of social web applications but still remain in control of my data. In the future I may move this to an even more replicated setup...

The basic idea is that the cloud is a trap that will move your data, and your applications beyond your control to proprietary data servers and web applications run by multinational corporations. If free software doesn't provide a compelling answer to that, we risk irrelevance.

A replicated, peer-to-peer system of synchronizing and sharing your data could be the answer. And Midgard2 is a framework providing just that. Bindings to different languages like PHP, Python and Mono, interprocess communications via D-Bus and XMPP, replication, and ability to run the same software from big server clusters to Nokia internet tablets should all help us get there.

In the conference we focused on outlining the big vision, and then ran a workshop where we showed some practical aspects of this. We set up Midgard2, and built a web application that allowed user to input a RSS feed address. This was stored to the database via midgard-php. Then a midgard-python process got notified about this via D-Bus, fetched the items and stored them to the database. The web front-end then displayed the articles. A clean example of interprocess communications.

Peer-to-peer replication we demonstrated in Ville Sundell's XMPP workshop where we built a Python replication daemon monitoring for database changes via D-Bus and transmitting them via Jabber to other Midgard boxes. Quite promising! But still many things need to be written before we are in the "your data everywhere" utopia...

Oh, and for those wondering: Midgard2 is very much GNOME software, running on top of GLib, libgda, D-Bus and so forth.

Can you see a change when former MidCOM lead developer Torben withdrew from the project?