Get your bundles back

>> Monday, December 06, 2010

I tried to update to the latest  integration build last week.  I couldn't because I had accidentally updated to a nightly build at some point. This problem wouldn't happen to most users because only the Eclipse and Indigo repositories are defined by default in the SDK.   I have many repositories defined in my install because I'm always testing bugs in different test repos.

Some background:

Eclipse bundles have four part version numbers. Major.minor.service.qualifier. With a nightly build,  the qualifiers of all our bundles is replaced with the build id.  For example, here's a subset of the bundles from a recent nightly build. Note the qualifier is always N20101201-2000.



With an integration build, the qualifier of the bundle reflects the CVS tag that the developer submitted to the  map file.  Looking at at subset of bundles from the latest integration build, note that the p2.garbagecollector bundle has a qualifier of v201011-2100. 


Since the first three segments of the bundle's version match, and N > v,  the bundles from the nightly build always sort higher. This caveat also extends to the IU of the product id, in my case, the org.eclipse.sdk product id.  Since I had accidentally upgraded to a nightly build, I couldn't update to any future integration builds because it's bundles will always sort lower.


No problem, I thought.  I'll just revert to an earlier install.  That's one of the many things that I love about p2.  Because p2 stores your previous profiles in the profile registry, you can revert to a previous state of your installation.

What's a profile? A profile contains properties defining the “environment” such as operating system, windowing system, architecture and install location.  It also contains the list of IUs and the properties associated with each IUs.  Artifacts are the actual content being installed (bytes).  Metadata consists of Installable units (IUs) that describes the content that can be installed.  Profiles are held by a profile registry which can store many profiles.

I tried to revert to a previous build.  Error- Cannot find bundles in repository.  Hmm, it appeared that p2 had garbage collected the old bundles in my install and was looking in my defined repositories for the bundles.   Ironically, I had recently cleaned up our integration builds repo so that we weren't consuming so much space on the eclipse.org servers. So the bundles I needed to revert to were nowhere to be found.

I didn't really want to roll back to an even earlier build. And I wanted to be able to update to newer integration builds.  (My eclipse install dates to March 2009 and I've been updating it since then without an issue). Since I'm the Eclipse project release engineer, my solution was to rerun the missing build locally and recreate those exact same bundles on my local linux server.  Reproducible builds are essential.  We tag the builder and map files each build so that the builds can be run again and create the same binary bits.  I wrote a little shell script to use the versions of the map files and builder that were use to create that build. Ran the build. Added the temporary repo to my install and tried to revert again.  I was feeling pretty confident that this would work, or to paraphrase xckd, "Watch out, I know release engineering". 



No.  Could not revert.  It turns out that I had also been using an  integration build of Mylyn which had also been cleaned from their repo.  So I had to revert to an earlier Eclipse milestone build before I had installed Mylyn, then update to the latest SDK I-build and re-install Mylyn.

In conclusion
  1. You can selectively update part of your install, but cannot selectively revert.  Reverting to a profile state is an all or nothing deal.
  2. Regular expressions are better than release engineers.

Further reading

OSGi semantic versioning
Eclipse Version Numbering
p2 tutorial
CRISP builds
p2 garbage collection

Read more...

Building Open Source: the Backstory

>> Friday, December 03, 2010

Last week, I wrote a post that compared the Eclipse and Mozilla build infrastructure.   It was surprised by the interest it generated.  My conclusion? People like to learn about complex build systems and scalability. Shiny hardware doesn't hurt either.  At the same time, I was also thinking about what would make an interesting proposal to submit to EclipseCon.  

In the midst of the mundane task of rebooting a performance machine in the lab, I was inspired.  Wouldn't it be interesting to open the door into a larger set of open source communities? To explore how they manage their build and release processes, hardware, software and the underlying technologies they use to transform source into binaries?


Image © criggchef, http://www.flickr.com/photos/criggchef/2665328223/   licensed under Creative Commons by-nc-sa 2.0


At Eclipse, sometimes we tend to limit ourselves to examining the projects within the Eclipse ecosystem itself as benchmarks for comparison.   Other open source foundations, such as Mozilla, Gnome, Linux and Apache, have similar challenges.   Do they have a centralized build farm?  How do they support multiple streams, platforms and SCMs? How do they manage hardware and parallelize enormous test suites? What CI engine do they use? Why? How do they optimize their build processes to support their community's goals?  I think the open source build backstory would expose some interesting ideas to the Eclipse community.



Image of  Toronto Reference Library © Andrew Louis, http://www.flickr.com/photos/hyfen/3562200168/  licensed under Creative Commons by-nc-sa 2.0

You may be thinking, "Kim, you're an Eclipse release engineer.  What do you know about builds at other open source foundations?"   The beautiful thing about open source is that it is full of passionate people who love to talk about their work. And by virtue of its open nature, we are free to discuss and learn from others.  I'm confident that I could create compelling content by talking to other release engineers to tell the story behind their builds. 




Image of Thomas Fisher Rare Books Library, Toronto © Andrew Louis, http://www.flickr.com/photos/hyfen/3562167028/ licensed under Creative Commons by-nc-sa 2.0

There are certainly a lot of fantastic talks proposed for EclipseCon this year.  If you're interested in this talk, please express your interest on the EclipseCon proposal. Thank you for your consideration!

Read more...

  © Blogger template Simple n' Sweet by Ourblogtemplates.com 2009

Back to TOP