Reminder: Release Engineering Special Issue submission deadline is August 1, 2014

>> Friday, July 18, 2014

Just a friendly reminder that the deadline for the Release Engineering Special Issue is August 1, 2014.  If you have any questions about the submission process or a topic that's you'd like to write about, the guest editors, including myself, are happy to help you!


Mozilla pushes - June 2014

Here's June 2014's  analysis of the pushes to our Mozilla development trees. You can load the data as an HTML page or as a json file

This was another record breaking month with a total of 12534 pushes.  As a note of interest, this is is over double the number of pushes we had in June 2013. So big kudos to everyone who helped us scale our infrastructure and tooling.  (Actually we had 6,433 pushes in April 2013 which would make this less than half because June 2013 was a bit of a dip.  But still impressive :-)

  • 12534 pushes
    • new record
  • 418 pushes/day (average)
    • new record
  • Highest number of pushes/day: 662 pushes on June 5, 2014
    • new record
  • Highest 23.17 pushes/hour (average)
    • new record

General Remarks
The introduction of Gaia-try in April has been very popular and comprised around 30% of pushes in June compared to 29% last month.
The Try branch itself consisted of around 38% of pushes.
The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 21% of all the pushes, compared to 22% in the previous month.

June 2014 was the month with most pushes (12534 pushes)
June 2014 has the highest pushes/day average with
418 pushes/day
June 2014 has the highest average of "pushes-per-hour" is
23.17 pushes/hour
June 4th, 2014 had the highest number of pushes in one day with
662 pushes


This week in Mozilla Releng - July 4, 2014

>> Friday, July 04, 2014

This is a special double issue of this week in releng. I was so busy in the last week that I didn't get a chance to post this last week.  Despite the fireworks for Canada Day and Independence Day,  Mozilla release engineering managed to close some bugs. 

Major highlights:

  • Armen, although he works on Ateam now, made blobber uploads discoverable and blogged about it.  Blobber is a server and client side set of tools that allow Releng's test infrastructure to upload files without requiring to deploy ssh keys on them. 
  • Callek and Coop, who served on buildduty during the past two weeks worked to address capacity issues with our test and build infrastructure.  We hit a record of 88,000 jobs yesterday which led to high pending counts.
  • Kim is trying to address the backlog of Android 2.3 test jobs  by moving more test jobs to AWS from our inhouse hardware now that Geoff on the Ateam has found a suitable image.
  • Rail switched jacuzzi EBS from magnetic to SSD.  Jacuzzis are similar pools of build machines  and switching their EBS storage from magnetic to SSD in AWS will improve build times.
 Completed work (resolution is 'FIXED'):
In progress work (unresolved and not assigned to nobody):


Introducing Mozilla Releng's summer interns

>> Friday, June 20, 2014

The Mozilla Release Engineering team recently welcomed three interns to our team for the summer.

Ian Connolly is a student at Trinity College in Dublin. This is his first term with Mozilla and he's working on preflight slave tasks and an example project for Releng API.
Andhad Jai Singh is a student at Indian Institute of Technology Hyderabad.  This is his second term working at Mozilla, he was a Google Summer of Code student with the Ateam last year.  This term he's working on generating partial updates on request.
John Zeller is also a returning student and studies at Oregon State University.  He previously had a work term with Mozilla releng and also worked during the past school term as a student worker implementing Mozilla Releng apps in Docker. This term he'll work on updating our ship-it application  so that release automation updates ship it more frequently so we can see the state of the release, as well as integrating post-release tasks.


View from Mozilla San Francisco Office

Please drop by and say hello to them if you're in our San Francisco office.  Or say hello to them in #releng - their irc nicknames are ianconnolly, ffledgling and zeller respectively.



This week in Mozilla Releng - June 20, 2014

Ben is away for the next few Fridays, so I'll be covering this blog post for the next couple of weeks.

Major highlights:

Completed work (resolution is 'FIXED'):
In progress work (unresolved and not assigned to nobody):


Talking about speaking up

>> Monday, June 09, 2014

We all interpret life through the lens of our previous experiences.  It's difficult to understand what each day is like for someone who has had a life fundamentally different from your own because you simply haven't had those experiences.  I don't understand what it's like to transition from male to female while involved in an open source community.  I don't know the steps taken to become an astrophysicist.  To embark to a new country as an immigrant.   I haven't lived struggled to survive on the streets as homeless person. Or a person who has been battered by domestic abuse.  To understand the experiences of others, all we can do is listen and learn from others, with empathy.

There have been many news stories recently about women or other underrepresented groups in technology.   I won't repeat them because frankly, they're quite depressing.  They go something like this:
1.  Incident of harassment/sexism either online/at a company/in a community/at a conference
2.  People call out this behaviour online and ask the organization to apologize and take steps to prevent this in the future.
3.  People from underrepresented groups who speak up about behaviour are told that their feelings are not valid or they are overreacting.  Even worse, they are harassed online with hateful statements telling them they don't belong in tech or are threatened with sexual assault or other acts of violence.
4.  Company/community/conference apologizes and issue written statement. Or not.
5. Goto 1

I watched an extraordinary talk the other day that really provided a vivid perspective about the challenges that women in technology face and what people can do to help. Brianna Wu is head of development at Giant Spacekat, a game development company.  She gave the talk "Nine ways to stop hurting and start helping women in tech" at AltConf last week.  She is brutally honest with the problems that exist in our companies and communities, and the steps forward to make it better. 

She talks about how she is threatened and harassed online. She also discusses how random people threatening you on the internet is not a just theoretical, but really frightening because she knows it could result in actual physical violence.   The same thing applies to street harassment. 

Here's the thing about being a woman.  I'm a physically strong person. I can run.  But I'm keenly aware that men are almost always bigger than me, and by basic tenets of physiology, stronger than me. So if a man tried to physically attack me, chances are I'd lose that fight.  So when someone threatens you, online or not, it is profoundly frightening because you fear for your physical safety. And to have that happen over and over again, like many women in our industry experience, apart from being terrifying, is exhausting and has a huge emotional toll.

I was going to summarize the points she brings up in her talk but she speaks so powerfully that all I can do is encourage you to watch the talk.

One of her final points really drives home the need for change in our industry when she says to the audience "This is not a problem that women can solve on their own....If you talk to your male friends out there, you guys have a tremendous amount of power as peers.  To talk to them and say, look dude this isn't okay.  You can't do this, you can't talk this way.  You need to think about this behaviour. You guys need to make a difference in a way that I can't."  Because when she talks about this behaviour to men, it often goes in one ear and out the next.  To be a ally in any sense of the word, you need to speak up.

THIS 1000x THIS.

Thank you Brianna for giving this talk.  I hope that when others see it they will gain some insight and feel some empathy on the challenges that women, and other underrepresented groups in the technology industry face.  And that you will all speak up too.

Further reading
Ashe Dryden's The 101-Level Reader: Books to Help You Better Understand Your Biases and the Lived Experiences of People                                                                                                           
Ashe Dryden Our most wicked problem


Mozilla pushes - May 2014

>> Monday, June 02, 2014

Here's May's monthly analysis of the pushes to our Mozilla development trees.  You can load the data as an HTML page or as a json file

This was a record breaking month where we overcame our previous record of 8100+ pushes with a record of 11000+ pushes this month.  Gaia-try, just created in April has become a popular branch with 29% of pushes.

  • 11711  pushes
    • New record
  • 378 pushes/day (average)
    • New record
  • Highest number of pushes/day: 613 pushes on May 29, 2014
    • New record
  • 22 pushes/hour (average)
    • New record
General Remarks
The introduction of Gaia-try in April has been very popular and comprised around 29% of pushes in May.  The Try branch itself consisted of around 38% of pushes.
The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 22% of all the pushes, compared to 30% in the previous month.

May 2014 was the month with most pushes (11711 pushes)
May 2014 has the highest pushes/day average with 378 pushes/day
May 2014 has the highest average of "pushes-per-hour" is 22 pushes/hour
May 29th, 2014 had the highest number of pushes in one day with 613 pushes

May 2014 is a record setting month, 11711 pushes!

Note that Gaia-try was added in April and has quickly become a high volume branch

I changed the format of this pie chart this month.  It seemed to be previously based on several months data, but not all data from the previous year.  So I changed it to be only based on the data from the current month which seemed more logical.


20 years on the web

>> Friday, May 16, 2014

Note: I started writing this a long time ago as part of #mynerdstory but never got around to finishing it until recently.  So I changed it a bit when I noticed it had been over 20 years since I first used the internet.

I found this picture the other day.  It's me on graduation day at Acadia,  twenty years ago this month.  A lot has changed since then.

In the picture, I'm in Carnegie Hall, where the Computer Science department had their labs, classrooms and offices. I'm sitting in front of a Sun workstation, which ran a early version of Mosaic.  I recall the first time I saw a web browser display a web page, I was awestruck.   I think it was NASA's web page.  My immediate reaction was that I wanted to work on that, to be on the web.

As a I've mentioned before, my Dad was a manager at a software and services firm in Halifax.  He brought home our first computer when I was 9.  Dad was always upgrading the computers or fixing them and I'd watch him and asked lots of questions about how the components connected together.  In junior high, I taught myself BASIC from the manual, wrote a bunch of simple programs, and played so many computer games that my dreams at night became pixelated.  When I was 16, I started working at my Dad's office doing clerical work during the school break.  One of my tasks was to run a series of commands to connect to BITNET via an accoustic coupler using Kermit and download support questions from their university customers.  I thought it was so magical that these computers that were so physically distant could connect and communicate.

In high school, I took computer science in grade 12 and we wrote programs in Pascal on Apple IIs.  My computer science teacher was very enthusiastic and welcoming.  He taught us sorting algorithms, and binary trees, and other advanced topics that weren't on the curriculum. Since he had such an interest he taught a lot of extra material.  Thanks Mr. B. 

When it was time to apply to university,  I didn't apply to computer science.  I don't know why, my grades were fine and I certainly had the background.  I really lacked self confidence that I could do it.  In retrospect, I would have been fine.  I enrolled at Acadia in their Bachelor of Business Administration program, probably because I liked reading the Globe and Mail.

I arrived on campus with a PC to write papers and do my accounting assignments.  The reason I had access to a computer was that the company my Dad worked for allowed their employees borrow a computer for home use for a year at a time, then return it.  Otherwise, they were prohibitively expensive at the time.  My third year of university I decided that I was better suited to computer science than business so started taking all my elective courses from the computer science faculty.  I still wanted to graduate on in four years so I didn't switch majors.  It was such a struggle to scrape together the money from part-time jobs and student loans to pay for four years of university, let alone six.

One of my part-time jobs was helping people in the university computer labs with questions and fixing problems.  Everything was very text based back then.  We used Archie to search for files, read books transcribed by the Gutenberg project and use uudecode to assemble pictures posted to Usenet groups.  I applied for a Unix account on the Sun system that only the Computer Science students had access to.   It was called dragon and the head sysadmin had a sig that said "don't flame me, I'm on dragon".  I loved learning all the obscure yet useful Unix commands.

My third year I had a 386 portable running Windows 3.1.  I carried this computer all over campus, plugging it in at the the student union centre and working on finance projects with my business school colleagues.  By my fourth year, they had installed Sun workstations in the Computer Science labs with Mosaic installed.   This was my first view of the world wide web.   It was beautiful.  The web held such promise.

I applied for 40 different jobs before I graduated from Acadia and was offered a job in Ottawa working for the IT department of Revenue Canada.  A ticket out of rural Nova Scotia! I didn't like my first job there that much but they paid for networking and operating system courses that I took at night.  I was able to move to a new job in a year and started being a sysadmin for their email servers that served 30,000 users.  It was a lot of fun and I learned a tremendous amount about networking, mail related protocols and operating systems.  I also spent a lot of time in various server rooms across Canada installing servers.  Always bring a sweater.

I left after a few years to work at Nortel as technical support for a telephony switch that offloaded internet traffic from voice switches to a dedicated switch.  Most internet traffic back then was via modem which were longer duration calls than most voice calls and caused traffic issues.  I took a lot of courses on telephony protocols, various Unix variants and networking. I traveled to several telco customers to help configure systems and demonstrate product features. More time in cold server rooms.

Shortly after Mr. Releng and I got married we moved to Athens, Georgia where he was completing his postdoc.  I found a great job as a sysadmin for the UGA's computer systems division.  The group provided FTP, electronic courseware and email services to the campus.  We also secured a lot of hacked Linux servers set up by unknowing graduate students in various departments.  When I started, I didn't know Linux very well so my manager just advised me to install Red Hat about 30 times and change the options every time, learn how to compile custom kernels and so on.  So that's what I did.  At that time you also had to compile Apache from source to include any modules such as ssl support, or different databases so I also had fun doing that. 

We used to do maintenance on the computer systems between 5 and 7am once a week.  Apparently not many students are awake at that hour.  I'd get up at 4am and drive in to the university in the early morning, the air heavy with the scent of Georgia pine and the ubiquitous humidity.  My manager M, always made a list the night before of what we had to do, how long it would take, and how long it would take to back the changes out.  His attention to detail and reluctance to ever go over the maintenance window has stayed with me over time. In fact, I'm still kind of a maintenance nerd, always figuring out how to conduct system maintenance in the least disruptive way to users.  The server room at UGA was huge and had been in operation since the 1960s.  The layers of cable under the tiles were an archeological record of the progress of cabling within the past forty years.  M typed on a DVORAK keyboard, and was one of the most knowledgeable people about all the Unix variants, and how they differed. If he found a bug in Emacs or any other open source software, he would just write a patch and submit it to their mailing list.  I thought that was very cool.

After Mr. Releng finished his postdoc, we moved back to Ottawa.  I got a job at a company called OTI as a sysadmin.  Shortly after joining, my colleague J said "We are going to release an open source project called Eclipse, are you interested in installing some servers for it?"  So I set up Bugzilla, CVS, mailman, nntp servers etc.  It was a lot of fun and the project became very popular and generated a lot of traffic.  A couple years later the Eclipse consortium became the Eclipse Foundation and all the infrastructure management moved there. 

I moved to the release engineering team at IBM and started working with S who taught me the fundamentals of release engineering.  We would spent many hours testing and implementing new features in the build, and test environment, and working with the development team to implement new functionality, since we used Eclipse bundles to build Eclipse.  I have written a lot about that before on my blog so I won't reiterate.  Needless to say, being paid to work full time in an open source community was a dream come true.

A couple of years ago, I moved to work at Mozilla.  And the 20 year old who looked Mosaic for the first time and saw the beauty and promise of the web, couldn't believe where she ended up almost 20 years later.

Many people didn't grow up with the privilege that I have, with access to computers at such a young age, and encouragement to pursue it as a career.  I thank all of you who I have worked with and learned so much from.  Lots still to learn and do!


Release Engineering Special Issue

A different type of mobile farm  ©Suzie Tremmel, Creative Commons by-nc-sa 2.0

Are you a release engineer with a great story to share?  Perhaps the ingenious way that you optimized your build scripts to reduce end to end build time?  Or how you optimized your cloud infrastructure to reduce your IT costs significantly?  How you integrated mobile testing into your continuous integration farm?  Or are you a researcher who would like to publish their latest research in a area related to release engineering?

If so, please consider submitting a report or paper to the first IEEE Release Engineering special issue.   Deadline for submissions is August 1, 2014 and the special issue will be published in the Spring of 2015.

IEEE Release Engineering Special Issue

If you have any questions about the process or the special issue in general, please reach out to any of the guest editors.  We're happy to help!

We're also conducting a roundtable interview with several people from the release engineering community in the issue.  This should raise some interesting insights given the different perspectives that people from organizations with large scale release engineering efforts bring to the table.


  © Blogger template Simple n' Sweet by 2009

Back to TOP