Tag Archives: openstack

CERN CentOS Dojo, part 3 of 4: Friday Dojo

On Friday, I attended the CentOS Dojo at CERN, in Meyrin Switzerland.

CentOS dojos are small(ish) gatherings of CentOS enthusiasts that happen all over the world. Each one has a different focus depending on where it is held and the people that plan and attend it.

You can read more about dojos HERE.

On Friday, we had roughly 60-70 people in attendance, in a great auditorium provided by CERN. We had 97 people registered, and 75% is pretty standard turnout for free-to-register events, so we were very pleased.

You can get a general idea of the size of the crowd in this video:

The full schedule of talks can be seen here: https://indico.cern.ch/event/649159/timetable/#20171020

There was an emphasis on large-scale computing, since that’s what CERN does. And the day started with an overview of the CERN cloud computing cluster. Every time I attend this talk (and I’ve seen it perhaps 6 times now) the numbers are bigger and more impressive.

CERN and Geneva

This time, they reported 279 thousands cores in their cluster. That’s a lot. And it’s all running RDO. This makes me insanely proud to be a small part of that endeavor.

Other presentations included reports from various SIGs. SIGs are Special Interest Groups within CentOS. This is where the work is done to develop projects on top of CentOS, including packaging, testing, and promotion of those projects. You can read more about the SIGs here: https://wiki.centos.org/SpecialInterestGroup

If you want to see your project distributed in the CentOS distro, a SIG is the way to make this happen. Drop by the centos-devel mailing list to propose a SIG or join an existing one.

The entire day was recorded, so watch this space for the videos and slides from the various presentations.

The CERN folks appeared very pleased with the day, and stated their intention to do the event again on an annual basis, if all works out. These things aren’t free to produce, of course (even though we strive to make them always free to attend) so if your organization is interested in sponsoring future dojos, please contact me. I’ll also be publishing a blog post over on seven.centos.org in the coming days about what’s involved in doing one of these events, in case you’d like to host one at your own facility..

CERN Centos Dojo, event report: 2 of 4 – CERN tours

(This post is the second in a series of four. They are gathered here.)

The second half of Thursday was where we got to geek out and tour various parts of CERN.

I was a physics minor in college, many years ago, and had studied not just CERN, but many of the actual pieces of equipment we got to tour, so this was a great privilege.

We started by touring the data center where the data from all of the various physics experiments is crunched into useful information and discoveries. This was amazing for a number of reasons.

From the professional side, CERN is the largest installation of RDO – the project I work with at work – that we know of. 279 thousand cores running RDO OpenStack.

For those not part of my geek world, that translates into hundreds of thousands of physical computers, arranged in racks, crunching data to unlock the secrets of the universe.

For those that are part of my geek world, you can understand why this was an exciting thing to see in person and walk through.

The full photo album is here, but I want to particularly show a couple of shots:

Visiting CERN

Here we have several members of the RDO and CentOS team standing in front of some of the systems that run RDO.

Visiting CERN

And here we have a photo that only a geek can love – this is the actual computer on which the very first website ran. Yes, boys and girls, that’s Tim Berners-Lee’s desktop computer from the very first days of the World Wide Web. It’s ok to be jealous.

There will also be some video over on my YouTube channel, but I haven’t yet had an opportunity to edit and post that stuff.

Next, we visited the exhibit about the Superconducting Super Collider, also known as the Large Hadron Collider. This was stuff that I studied in college, and have geeked out about for the years since then.

There are pictures from this in the larger album, but I want to point out one particular picture of something that absolutely blew my mind.

Most of the experiments in the LHC involve accelerating sub-atomic particles (mostly protons) to very high speeds – very close to the speed of light – and then crashing them into something. When this happens, bits of it fly off in random directions, and the equipment has to detect those bits and learn things about them – their mass, speed, momentum, and so on.

In the early days, one of the the ways that they did this was to build a large chamber and string very fine wires across it, so that when the particles hit those wires it would cause electrical impulses.

Those electrical impulses were captured by these:

CERN visit

Those are individual circuit boards. THOUSANDS of them, each individually hand-soldered. Those are individual resistors, capacitors, and ICs, individually soldered to boards. The amount of work involved – the dedication, time, and attention to detail – is simply staggering. This photo is perhaps 1/1000th of the total number of boards. If you’ve done any hand-soldering or electronic projects, you’ll have a small sense of the scale of this thing. I was absolutely staggered by this device.

Outside on the lawn were several pieces of gigantic equipment that were used in the very early days of particle physics, and this was like having the pages of my college text book there in front of me. I think my colleagues thought I’d lost my mind a little.

College was a long time ago, and most of the stuff I learned has gone away, but I still have the sense of awe of it all. That an idea (let’s smash protons together!) resulted in this stuff – and more than 10,000 people working in one place to make it happen, is really a testament to the power of the human mind. I know some of my colleagues were bored by it all, but I am still reeling a little from being there, and seeing and touching these things. I am so grateful to Tim Bell and Thomas Oulevey for making this astonishing opportunity available to me.

Finally, we visited the ATLAS experiment, where they have turned the control room into a fish tank where you can watch the scientists at work.

CERN visit

What struck me particularly here was that most of the people in the room were so young. I hope they have a sense of the amazing opportunity that they have here. I expect that a lot of these kids will go on to change the world in ways that we haven’t even thought of yet. I am immensely jealous of them.

So, that was the geek chapter of our visit. Please read the rest of the series for the whole story.

CERN Centos Dojo 2017, Event report (0 of 4)

For the last few days I’ve been in Geneva for the CentOS dojo at CERN.

What’s CERN? – http://cern.ch/

What’s a dojo? – https://wiki.centos.org/Events/Dojo/

What’s CentOS? – http://centos.org/

A lot has happened that I want to write about, so I’ll be breaking this into several posts:

(As usual, if you’re attempting to follow along on Facebook, you’ll be missing all of the photos and videos, so you’ll really want to go directly to my blog, at http://drbacchus.com/)

 

Event report: OpenStack PTG

Last week I attended the second OpenStack PTG, in Denver. The first one was held in Atlanta back in February.

This is not an event for everyone, and isn’t your standard conference. It’s a working meeting – a developers’ summit at which the next release of the OpenStack software is planned. The website is pretty blunt about who should, and should not, attend. So don’t sign up without knowing what your purpose is there, or you’ll spend a lot of time wondering why you’re there.

I went to do the second installment of my video series, interviewing the various project teams about what they did in the just-released version, and what they anticipate coming in the next release.

The first of these, at the PTG in Atlanta, featured only Red Hat engineers. (Those videos are HERE.) However, after reflection, I decided that it would be best to not limit it, but to expand it to the entire community, focusing on cross-company collaboration, and trying to get as many projects represented as I could.

So, in Denver I asked the various project PTL (project technical leads) to do an interview, or to assemble a group of people from the project to do interviews. I did 22 interviews, and I’m publishing those on the RDO YouTube channel – http://youtube.com/RDOCommunity – just as fast as I can get them edited.

I also hosted an RDO get-together to celebrate the Pike release, and we had just over 60 people show up for that. Thank you all so much for attending! (Photos and video from that coming soon to a blog near you!)

So, watch my YouTube channel, and hopefully by the end of next week I’ll have all of those posted.

I love working with the OpenStack community because they remind me of Open Source in the old days, when developers cared about the project and the community at least as much, and often more, than about the company that happens to pay their paycheck. It’s very inspiring to listen to these brilliant men and women talking about their projects.

Event report: Red Hat Summit, OpenStack Summit

Event report: Red Hat Summit, OpenStack Summit

May 1-5, 2017 and May 8-11, 2017

During the first two weeks of May, I attended Red Hat Summit, followed by OpenStack Summit. Since both events were in Boston (although not at the same venue), many aspects of them have run together.

Mini-cluster

On the first day of Red Hat Summit, I received the mini-cluster, which had been built in Brno for the April Brno open house. There were one or two steps missing from the setup instructions, so with a great deal of help from Hugh Brock, it too most of the first day to get the cluster running. We’ll be publishing more details about the mini-cluster on the RDO blog in the next week or two. However, most of the problems were 1) it was physically connected incorrectly (ie, my fault) and 2) there were some routing table changes that were apparently not saved after initial setup.

Once the cluster was up, we connected to the ManageIQ cluster on the other side of our booth, and they were able to manage our OpenStack deployment. Thus, we were able to demonstrate the two projects working together.

In future events, we’d like to bring more projects into this arrangement – say, use Ceph for storage, or have ManageIQ managing OpenStack and oVirt, for example.

After we got the cluster working, in subsequent days, we just had to power it on, follow the startup instructions, and be patient. Again, more details of this will be in the RDO blog post in the coming  weeks.

Upcoming CentOS Dojos

I had conversations with two groups about planning upcoming CentOS Dojos.

The first of these will be at Oak Ridge National Labs (ORNL), and is
now tentatively scheduled for the first Tuesday in September.  (If you saw my internal event report, I mentioned July/August. This has since changed.) They’re interested in doing a gathering that would be about both CentOS and OpenStack, and draw together some of the local developer community. This will be held in conjunction with the local LOPSA group.

The second Dojo that we’re planning will be at CERN, where we have a great relationship with the cloud computing group, who run what we believe to be the largest RDO installation in the world. We have a tentative date of October 20th, immediately before Open Source Summit in Prague to make it easier to combine two trips for those traveling internationally. This event, too, would cover CentOS topics as well as OpenStack/RDO topics.

If you’re interested in participating in either one of these events, you need to be on the centos-promo mailing list. Send mail to centos-promo-subscribe@centos.org to subscribe, or visit https://lists.centos.org/mailman/listinfo/centos-promo for the
clicky-clicky version.

General Impressions

The Community Central area at Red Hat Summit was awesome. Sharing center stage with the product booths was a big win for our upstream first message, and we had a ton of great conversations with people who grasped the “X is the upstream for Red Hat X” concept, seemingly, for the first time. The “The Roots Are In The Community” posters resonated with a lot of people, so huge thanks to Tigert for pulling those together at the last minute.

The collaboration between RDO and ManageIQ was very rewarding, and helped promote the CloudForms message even more, because people could see it in action, and see how the communities work together for the greater good of humanity. I look forward to expanding this collaboration to all of the projects in the Community Central area by next year.

The space for Red Hat Summit was huge, making the crowd seem a lot smaller than it actually was. The opposite was true for  OpenStack Summit, where it was always crowded and seemed very busy, even though the crowd was smaller than last year.

 

Where next?

In three weeks I’ll be heading to the High Performance Computing event in Frankfurt. My mission there is to talk with people that are using CentOS and RDO in HPC, and collect user stories.

3 ways you find the right type of contributor and where to find them

Another one of Stormy’s questions caught my eye:

“What are 3 ways you find the right type of contributor, and where do you find them?”

Thinking back to the last few years of work on RDO, several answers come to mind:

The people asking the questions

Watch the traffic on your mailing list(s) and on the various support forums. The people that are asking the most questions are often great potential contributors. They are using the project, so they are invested in its success. They are experiencing problems with it, so they know where there are problems that need to be addressed. And they are outspoken enough, or brave enough, to talk about their difficulties publicly, so they are likely to be just as willing to talk about their solutions.

These are usually good people to approach ask ask to write about their user experience. This can often be done collaboratively, combining their questions with the eventual answers that they encountered.

If they appear to be the type of person who is implementing solutions to their problems, ask them to bring those solutions back to your community for inclusion in the code.

In RDO, people that participate in the rdo-list mailing list will sometimes end up contributing their solutions to the project, and eventually becoming regular contributors. We probably need to do a better job of encouraging them to do this, rather than just hoping it’s going to happen on its own.

The people answering the questions

In watching the question-and-answer on ask.openstack.org I often see names I don’t recognize, answering questions related to RDO. Sometimes these are Red Hat engineers who have recently joined the project, and I haven’t met yet. But sometimes, it’s people from outside of Red Hat who have developed expertise on RDO and are now starting to pay that back.

These are the people that I then try to approach via the private messaging feature of that website, and ask them what their story is. This occasionally evolves into a conversation that brings them to more active involvement in the project.

Most people like to be asked, and so asking someone specifically if they’d be willing to come hang out in the IRC channel, or answer questions on the mailing list, tends to be fairly effective, and is a good step towards getting them more involved in the project.

The people complaining

This a tricky one. People complain because something is broken, or doesn’t work as they expect it to. The traditional response to this in the free software world is “Patches Welcome.” This is shorthand for “Fix it yourself and stop bugging me.”

The trick here is to recognize that people usually take the trouble to complain because they want it to work and they’re looking for help. This passion to just make things work can often be harnessed into contributions to the project, if these people are treated with patience and respect.

Patience, because they’re already frustrated, and responding with frustration or defensiveness is just going to make them angry.

Respect, because they are trying to solve an actual real world problem, and they’re not just there to hassle you.

It’s important that you fix their problem. Or at least, try to move them towards that solution.

Once the problem is addressed, ask them to stay. Ask them to write about their situation, and the fix to it. Ask them to stick around and answer other questions, since they have demonstrated that they care about the project, at least to the point of getting it working.

When people complain about your project, you also have a great opportunity to brush them off and persuade them that you are uncaring and unwelcoming, which they will then go tell all of their friends and Twitter followers. This can be a very expensive thing to do, for your community. Don’t do that.

When people come to #rdo on Freenode IRC to ask their RDO and OpenStack questions, I frequently (usually) don’t know the answer myself, but I try to make an effort to connect them with the people that do know the answer. Fortunately, we have an awesome community, and once you bring a question to the attention of the right person, they will usually see it through to the right solution.

What’s new in OpenStack Ocata

OpenStack Ocata has now been out for a little over a month – https://releases.openstack.org/ – and we’re about to see the first milestone of the Pike release. Past cycles show that now’s about the time when people start looking at the new release to see if they should consider moving to it. So here’s a quick overview of what’s new in this release.

First, it’s important to remember that the Ocata cycle was very short. We usually do a release every six months, but with the rescheduling of the OpenStack Summit and OpenStack PTG (Project Team Gathering) events, Ocata was squeezed into 4 months to realign the releases with these events. So, while some projects squeezed a surprising amount of work into that time, most projects spent the time on smaller features, and finishing up tasks leftover from the previous release.

At a high level, the Ocata release was all about upgrades and containers – themes that I heard from almost every team I interviewed. (For the full interview series, see https://goo.gl/3aCQ2d ) How can we make upgrades smoother, and how can we deploy bits of the infrastructure in containers. These two things are closely related, and there seems to be more cross-project collaboration this time around than I’ve noticed in the past.

And the themes of upgrades and containers will continue to be prominent in the Pike cycle.

Highlights in the Ocata cycle include:

Auto-healing: Work was done in Heat to make it easier to recover from service failure. When an outage is detected, you can have Heat automatically spin up a replacement service, and swap it out without any intervention on the part of the operator.

Composability: Composable roles are a feature whereby you can specify details of how things are deployed, rather than allowing OpenStack to choose. You can, for example, specify that a particular hardware configuration be used for particular services. This is termed Composable Roles. Work was done in Ocata to expand this to composable upgrades, so that these roles are respected across upgrades as well.

Multi-factor auth in Keystone: Work was done in Keystone to improve support of MFA, including OTP (One Time Password) support, and per-user token expiration rules.

NFV: Network Function Virtualization continues to be an area where we’re seeing a lot of activity, and so a lot of the work in Nova, Neutron, and various other projects focuses on these developments. NFV has become more stable in this release, and is more fully integrated into TripleO for ease of deployment. This effort is happening under the Apex project – https://wiki.opnfv.org/display/apex/Apex

Upgrades: Upgrades were a common theme across all projects, with the emphasis being the ability to upgrade from one release to the next with as close to zero downtime as possible. Much of this work centers around TripleO, Heat, and Mistral, for orchestration and automation of the process.

Containers: While centered around the Kolla project, containerization was a theme in many of the projects this cycle. The eventual goal, at least according to some, is that OpenStack services will be deployed in containers by default by the Pike release. This of course poses a real challenge for the Ocata -> Pike upgrade path (migrating from non-container to container in the course of the upgrade), and that’s something that the TripleO people are working hard on.

Security: TLS-everywhere made strides forward in Ocata, with connections between services moving to TLS. This involves changes to Barbican as well, for key management for the shared keys between services, to ensure that your traffic is secure between components of your cloud, which may be located in different data centers around the world.

Collaboration: Something I heard more this year than in previous years was talk of collaboration between projects. This has, of course, always been happening. However at the PTG in Atlanta, it was a major focus, with time set aside for cross-project meetings focusing on the interface between one service and another. I also heard from several people that the PTG allowed a focus, and a camaraderie, that was not possible when the design summit was part of OpenStack Summit. This resulted in fewer interpersonal tensions, and a lot more work getting done.

Everything else: The difficulty with OpenStack is that it’s just so big. While these are the things that stood out to me, someone else is likely to pull out different highlights, depending on their interests. So I encourage you to look at some of the other “What’s new in Ocata” articles out there, including especially “53 new things to look for in OpenStack Ocata” – https://www.mirantis.com/blog/53-new-things-to-look-for-in-openstack-ocata/ and, if you have a lot of time, or have interest in a particular project, check out the official release notes – https://releases.openstack.org/ocata/  And take a moment to watch my interviews with various Red Hat OpenStack engineers, from the Atlanta PTG, here: https://goo.gl/3aCQ2d.

OpenStack PTG, trip report

last week, I attended the OpenStack PTG (Project Teams Gathering) in Atlanta.

Even more in depth: PTG info at https://www.openstack.org/blog/2016/05/faq-evolving-the-openstack-design-summit/

TL;DR:

1) This is a hugely productive event, with project teams getting an enormous amount of work done without the distractions that are usually present at a conference.

2) I remain very concerned about how this event will effect the
character of OpenStack Summit – removing the bulk of the engineers from that event, and making it more product/marketing/sales focused. Time will tell

At the gathering, I did 23 interviews with Red Hat engineers about what they did in the Ocata release. You can see some of those interview on the RDO YouTube Channel. I’m not done editing them all yet, but they will appear over the coming weeks as part of various blog posts, as well as all of them appearing in that YouTube playlist.

I am constantly blown away by the passion, expertise, and
professionalism of the folks I get to work with. Wow.

Anyways, more about the PTG.

I was (and, really, still am) very skeptical about this new format.
Splitting OpenStack Summit into four events rather than two has already had significant impact on travel budgets, not just at Red Hat, but also at other companies involved in OpenStack. A lot of companies, for example, didn’t send anyone to FOSDEM, and we had a hard time staffing the OpenStack table there. Usually people work one shift at the table, but this year several of us worked 4 and 5 shifts to cover all the slots.

I am concerned that splitting the engineers off into their own event
will significantly change the character of OpenStack Summit from being a community-centric, tech-centric event, to more of a sales and marketing event, light on technical depth.

But this event, for what was intended, has already been amazing.
Everyone is commenting on how much is getting done, how much less distracted the team meetings are, how much better the teams are gelling than they have at any previous event. This is a working event, and people are here to get work done. They are meeting all day, every day, working on plans and blueprints, and fighting out agreements on things that would take weeks in email, and everyone seems VERY pleased with outcomes.

So, perhaps the trade off will be worth it. Time will tell. Regardless, Erin Disney and her team put on an amazing event that fulfilled, and exceeded, its goals.

On Wednesday  night, everyone that has ever contributed a patch to RDO was invited for drinks and hors d’oeuvres at the SideBar, and while there the RDO Ocata release announcement was sent out.

We had about 50 people in attendance, who ate and drank up all of my budget in about 2 hours.

Here’s some pictures.

 

Project Leader

I was recently asked to write something about the project that I work on – RDO – and one of the questions that was asked was:

A healthy project has a visible lead(s). Who is the project lead(s) for this project?

This struck me as a strange question because, for the most part, the open source projects that I choose to work on don’t have a project lead, but are, rather, led by community consensus, as well as a healthy dose of “Just Do It”. This is also the case with RDO, where decisions are discussed in public on the mailing list, and on IRC meetings, and those that step up to do the work have more practical influence than those that just talk about it.

Now, this isn’t to say that nobody takes leadership or ownership of the projects. In many senses, everyone does. But, of course, certain people do rise to prominence from time to time, just based on the volume of work that they do, and these people are the de facto leaders for that moment.

There’s a lot of different leadership styles in open source, and a lot of projects do in fact choose to have one technical leader who has the final say on all contributions. That model can work well, and does in many cases. But I think it’s important for a project to ask itself a few questions:

  • What do we do when a significant number of the community disagrees with the direction that this leader is taking things?
  • What happens when the leader leaves? This can happen for many different reasons, from vacation time, to losing interest in the project, to death.
  • What do we do when the project grows in scope to the point that a single leader can no longer be an expert on everything?

A strong leader who cares about their project and community will be able to delegate, and designate replacements, to address these concerns. A leader who is more concerned with power or ego than with the needs of their community is likely to fail on one or more of these tests.

But, I find that I greatly prefer projects where project governance is of the people, by the people, and for the people.

OpenStack Summit Barcelona, 3 of N

Continuing the saga of OpenStack Summit Barcelona …

Wednesday was a very long day – about 10 hours working the Red Hat booth, with hundreds and hundreds of people dropping by to chat, get tshirts, ask questions about our various products, and just hang out.

While Tuesday was “day two”, Wednesday is the second full day of the expo hall, and so this is the time when you get even less frantic swag-seeking, and more people looking for information. Which is good, because we were starting to run out of the swag.

What stood out on Wednesday was the sheer number of people who stopped by just to say how awesome RDO is, and how much they appreciate the work that the RDO community does. Several people also asked how they could get involved in making things happen that are not happening yet. So, this was very encouraging.

Wednesday evening was a quiet dinner-then-bed evening, as is usually the case by this point in a conference, when the true exhaustion of long days and jet lag really catches up. I had dinner with a small group of friends and coworkers, which was really delicious, but I have no idea how to tell you what I had. 🙂

And Thursday was the final day. We got the expo hall open time wrong, so I was actually there in the booth for 2 hours before it opened. Gah! And then, another really long day of answering questions, and giving away the last of the ducks.

img_20161030_154636The duck tradition that was started in Paris continues, and we plan to have more ducks in Boston. We only had 800 ducks in Barcelona, because I waited kind of late to order, and the vendor was low on inventory. Any suggestions of how the duck should dress for Boston?

We closed up just after 4, packed everything up, and I headed to the beach. Unfortunately, right about the time I got there, it started to get overcast and windy, so I only got about a half hour on the beach before I gave up and left, shivering. Oh, well, better than nothing.

Thursday night was another dinner-and-fall-in-bed-early evening, as I was completely exhausted. Unfortunately, as usual when I have a flight in the morning, I didn’t get much sleep. I have this bad habit of waking up every few hours to check the clock, even though I have several alarms set. Some day, I hope to get over that.

On Friday, I shared a cab to the airport with Jen, and we boarded the flight for JFK.

It’s good to be home.