Tag Archives: rdo

CERN CentOS Dojo, part 3 of 4: Friday Dojo

On Friday, I attended the CentOS Dojo at CERN, in Meyrin Switzerland.

CentOS dojos are small(ish) gatherings of CentOS enthusiasts that happen all over the world. Each one has a different focus depending on where it is held and the people that plan and attend it.

You can read more about dojos HERE.

On Friday, we had roughly 60-70 people in attendance, in a great auditorium provided by CERN. We had 97 people registered, and 75% is pretty standard turnout for free-to-register events, so we were very pleased.

You can get a general idea of the size of the crowd in this video:

The full schedule of talks can be seen here: https://indico.cern.ch/event/649159/timetable/#20171020

There was an emphasis on large-scale computing, since that’s what CERN does. And the day started with an overview of the CERN cloud computing cluster. Every time I attend this talk (and I’ve seen it perhaps 6 times now) the numbers are bigger and more impressive.

CERN and Geneva

This time, they reported 279 thousands cores in their cluster. That’s a lot. And it’s all running RDO. This makes me insanely proud to be a small part of that endeavor.

Other presentations included reports from various SIGs. SIGs are Special Interest Groups within CentOS. This is where the work is done to develop projects on top of CentOS, including packaging, testing, and promotion of those projects. You can read more about the SIGs here: https://wiki.centos.org/SpecialInterestGroup

If you want to see your project distributed in the CentOS distro, a SIG is the way to make this happen. Drop by the centos-devel mailing list to propose a SIG or join an existing one.

The entire day was recorded, so watch this space for the videos and slides from the various presentations.

The CERN folks appeared very pleased with the day, and stated their intention to do the event again on an annual basis, if all works out. These things aren’t free to produce, of course (even though we strive to make them always free to attend) so if your organization is interested in sponsoring future dojos, please contact me. I’ll also be publishing a blog post over on seven.centos.org in the coming days about what’s involved in doing one of these events, in case you’d like to host one at your own facility..

CERN Centos Dojo, event report: 2 of 4 – CERN tours

(This post is the second in a series of four. They are gathered here.)

The second half of Thursday was where we got to geek out and tour various parts of CERN.

I was a physics minor in college, many years ago, and had studied not just CERN, but many of the actual pieces of equipment we got to tour, so this was a great privilege.

We started by touring the data center where the data from all of the various physics experiments is crunched into useful information and discoveries. This was amazing for a number of reasons.

From the professional side, CERN is the largest installation of RDO – the project I work with at work – that we know of. 279 thousand cores running RDO OpenStack.

For those not part of my geek world, that translates into hundreds of thousands of physical computers, arranged in racks, crunching data to unlock the secrets of the universe.

For those that are part of my geek world, you can understand why this was an exciting thing to see in person and walk through.

The full photo album is here, but I want to particularly show a couple of shots:

Visiting CERN

Here we have several members of the RDO and CentOS team standing in front of some of the systems that run RDO.

Visiting CERN

And here we have a photo that only a geek can love – this is the actual computer on which the very first website ran. Yes, boys and girls, that’s Tim Berners-Lee’s desktop computer from the very first days of the World Wide Web. It’s ok to be jealous.

There will also be some video over on my YouTube channel, but I haven’t yet had an opportunity to edit and post that stuff.

Next, we visited the exhibit about the Superconducting Super Collider, also known as the Large Hadron Collider. This was stuff that I studied in college, and have geeked out about for the years since then.

There are pictures from this in the larger album, but I want to point out one particular picture of something that absolutely blew my mind.

Most of the experiments in the LHC involve accelerating sub-atomic particles (mostly protons) to very high speeds – very close to the speed of light – and then crashing them into something. When this happens, bits of it fly off in random directions, and the equipment has to detect those bits and learn things about them – their mass, speed, momentum, and so on.

In the early days, one of the the ways that they did this was to build a large chamber and string very fine wires across it, so that when the particles hit those wires it would cause electrical impulses.

Those electrical impulses were captured by these:

CERN visit

Those are individual circuit boards. THOUSANDS of them, each individually hand-soldered. Those are individual resistors, capacitors, and ICs, individually soldered to boards. The amount of work involved – the dedication, time, and attention to detail – is simply staggering. This photo is perhaps 1/1000th of the total number of boards. If you’ve done any hand-soldering or electronic projects, you’ll have a small sense of the scale of this thing. I was absolutely staggered by this device.

Outside on the lawn were several pieces of gigantic equipment that were used in the very early days of particle physics, and this was like having the pages of my college text book there in front of me. I think my colleagues thought I’d lost my mind a little.

College was a long time ago, and most of the stuff I learned has gone away, but I still have the sense of awe of it all. That an idea (let’s smash protons together!) resulted in this stuff – and more than 10,000 people working in one place to make it happen, is really a testament to the power of the human mind. I know some of my colleagues were bored by it all, but I am still reeling a little from being there, and seeing and touching these things. I am so grateful to Tim Bell and Thomas Oulevey for making this astonishing opportunity available to me.

Finally, we visited the ATLAS experiment, where they have turned the control room into a fish tank where you can watch the scientists at work.

CERN visit

What struck me particularly here was that most of the people in the room were so young. I hope they have a sense of the amazing opportunity that they have here. I expect that a lot of these kids will go on to change the world in ways that we haven’t even thought of yet. I am immensely jealous of them.

So, that was the geek chapter of our visit. Please read the rest of the series for the whole story.

CERN Centos Dojo 2017, Event report (0 of 4)

For the last few days I’ve been in Geneva for the CentOS dojo at CERN.

What’s CERN? – http://cern.ch/

What’s a dojo? – https://wiki.centos.org/Events/Dojo/

What’s CentOS? – http://centos.org/

A lot has happened that I want to write about, so I’ll be breaking this into several posts:

(As usual, if you’re attempting to follow along on Facebook, you’ll be missing all of the photos and videos, so you’ll really want to go directly to my blog, at http://drbacchus.com/)

 

Event report: OpenStack PTG

Last week I attended the second OpenStack PTG, in Denver. The first one was held in Atlanta back in February.

This is not an event for everyone, and isn’t your standard conference. It’s a working meeting – a developers’ summit at which the next release of the OpenStack software is planned. The website is pretty blunt about who should, and should not, attend. So don’t sign up without knowing what your purpose is there, or you’ll spend a lot of time wondering why you’re there.

I went to do the second installment of my video series, interviewing the various project teams about what they did in the just-released version, and what they anticipate coming in the next release.

The first of these, at the PTG in Atlanta, featured only Red Hat engineers. (Those videos are HERE.) However, after reflection, I decided that it would be best to not limit it, but to expand it to the entire community, focusing on cross-company collaboration, and trying to get as many projects represented as I could.

So, in Denver I asked the various project PTL (project technical leads) to do an interview, or to assemble a group of people from the project to do interviews. I did 22 interviews, and I’m publishing those on the RDO YouTube channel – http://youtube.com/RDOCommunity – just as fast as I can get them edited.

I also hosted an RDO get-together to celebrate the Pike release, and we had just over 60 people show up for that. Thank you all so much for attending! (Photos and video from that coming soon to a blog near you!)

So, watch my YouTube channel, and hopefully by the end of next week I’ll have all of those posted.

I love working with the OpenStack community because they remind me of Open Source in the old days, when developers cared about the project and the community at least as much, and often more, than about the company that happens to pay their paycheck. It’s very inspiring to listen to these brilliant men and women talking about their projects.

3 ways you find the right type of contributor and where to find them

Another one of Stormy’s questions caught my eye:

“What are 3 ways you find the right type of contributor, and where do you find them?”

Thinking back to the last few years of work on RDO, several answers come to mind:

The people asking the questions

Watch the traffic on your mailing list(s) and on the various support forums. The people that are asking the most questions are often great potential contributors. They are using the project, so they are invested in its success. They are experiencing problems with it, so they know where there are problems that need to be addressed. And they are outspoken enough, or brave enough, to talk about their difficulties publicly, so they are likely to be just as willing to talk about their solutions.

These are usually good people to approach ask ask to write about their user experience. This can often be done collaboratively, combining their questions with the eventual answers that they encountered.

If they appear to be the type of person who is implementing solutions to their problems, ask them to bring those solutions back to your community for inclusion in the code.

In RDO, people that participate in the rdo-list mailing list will sometimes end up contributing their solutions to the project, and eventually becoming regular contributors. We probably need to do a better job of encouraging them to do this, rather than just hoping it’s going to happen on its own.

The people answering the questions

In watching the question-and-answer on ask.openstack.org I often see names I don’t recognize, answering questions related to RDO. Sometimes these are Red Hat engineers who have recently joined the project, and I haven’t met yet. But sometimes, it’s people from outside of Red Hat who have developed expertise on RDO and are now starting to pay that back.

These are the people that I then try to approach via the private messaging feature of that website, and ask them what their story is. This occasionally evolves into a conversation that brings them to more active involvement in the project.

Most people like to be asked, and so asking someone specifically if they’d be willing to come hang out in the IRC channel, or answer questions on the mailing list, tends to be fairly effective, and is a good step towards getting them more involved in the project.

The people complaining

This a tricky one. People complain because something is broken, or doesn’t work as they expect it to. The traditional response to this in the free software world is “Patches Welcome.” This is shorthand for “Fix it yourself and stop bugging me.”

The trick here is to recognize that people usually take the trouble to complain because they want it to work and they’re looking for help. This passion to just make things work can often be harnessed into contributions to the project, if these people are treated with patience and respect.

Patience, because they’re already frustrated, and responding with frustration or defensiveness is just going to make them angry.

Respect, because they are trying to solve an actual real world problem, and they’re not just there to hassle you.

It’s important that you fix their problem. Or at least, try to move them towards that solution.

Once the problem is addressed, ask them to stay. Ask them to write about their situation, and the fix to it. Ask them to stick around and answer other questions, since they have demonstrated that they care about the project, at least to the point of getting it working.

When people complain about your project, you also have a great opportunity to brush them off and persuade them that you are uncaring and unwelcoming, which they will then go tell all of their friends and Twitter followers. This can be a very expensive thing to do, for your community. Don’t do that.

When people come to #rdo on Freenode IRC to ask their RDO and OpenStack questions, I frequently (usually) don’t know the answer myself, but I try to make an effort to connect them with the people that do know the answer. Fortunately, we have an awesome community, and once you bring a question to the attention of the right person, they will usually see it through to the right solution.

OpenStack Summit, Barcelona, 2 of n

Tuesday, the first day of the main event, was, as always, very busy. I spent most of the day working the Red Hat booth. We started at 10 setting up, and the mob came in around 10:45.

Day two of booth duty is always interesting, because it’s after the swag feeding frenzy has died down a bit that you start hearing from the people that actually care about what you’re “selling”. You get the questions. And what’s been fascinating in the 6 summits I’ve attended is that the bar has been raised a LOT on the questions. In Hong Kong, my first Summit, there were still a lot of people asking what OpenStack was, and nobody had any idea what RDO was. Now, the questions are about specific deployment scenarios, projects that aren’t yet being packaged, the future of TripleO, and so on, with only a handful of people asking what RDO is.

OpenStack has clearly made the transition from “something to consider some day” to “of course we are, and what are you going to do to make it better?”

Another awesome improvement this Summit was how the RDO community stepped up to help in the booth. Every single hour of the day, I had at least one, and usually two, members of the RDO community in the booth with me, either doing an “Ask Me Anything About RDO”, or doing some kind of a demo. It was *awesome*. Maybe next year, I’ll just stay home. 😉

The highlight of the day was the RDO/Ceph community meetup. We had 4 hours at the Gym Bar in the Princess Hotel.

Members of the Ceph and RDO community presented, lightning talk style (5 minute presentations) on a variety of topic. Speakers were threatened with being thrown in the pool if they went over 5 minutes, but we managed to restrain ourselves.

By the end, we had checked in 215 people overall, and we had 12 speakers. The food was good. The speakers were awesome. The only complaint was that the people not actually listening to the talks would NOT shut up, so it was hard to hear. Eventually, one of the speakers shouted at them to shut up or get out, and most of them moved to the other side of the room.

I have a recording of the event, but I don’t expect it’s going to be usable, due to the noise level. I haven’t had a chance to review it yet. Next week, for sure. I also hope to have (some of?) the presentation slides from the various speakers posted somewhere. Watch rdo-list and/or @rdocommunity for details.

After the talks were over, we had an hour or so left, and I cowardly skipped out. There comes a time when I have just had too much social interaction, and I need quiet time.

So, that was Tuesday. Another success, and another day to be glad that I work with such an awesome community.

 

OpenStack Summit, Barcelona, 1 of n

I have the best intentions of blogging every day of an event. But every day is always so full, from morning until the time I drop into bed exhausted.

I used to imagine wandering around the world like Hemingway, seeing exotic places, and writing witty stories about the interesting people I met. I have the great good fortune to have the traveling as part of my job. If only I could find the time for the stories.

Anyways … I’m in Barcelona for OpenStack Summit. This is always an impressive, and somewhat overwhelming, event, with 7000+ people attending, dozens of companies presenting their various products, hundreds of technical presentations, and after-hour events every evening.

The photos are on flickr, at https://www.flickr.com/photos/rbowen/albums/72157675620637236

On Sunday, I met up with Jen and various other members of the events team to scope out the venue. We walked over to the Princess Hotel where a number of our meetings and social events are to be held. And we walked down to El Boo, where the employee party was to be held. Both venues were just great.

Later in the day I visited Sagrada Familia, a cathedral which has been under construction since 1884, on and off, and isn’t done yet. It was weird and improbable, and fascinating, and beautiful. I think the architect’s driving passion was to be different from anything you’ve ever seen before.

I also spent a little time on the lovely beach, Mar Bella, which is right in front of my hotel.

On Monday, we had the RDO Infrastructure gathering in one of the rooms at the Princess. we had about 20 people in attendance, and made good progress on a number of issues. The Ocata cycle is going to see more improvements to how RDO works.  More details on this meeting coming to the RDO mailing list soon.

Although the conference didn’t officially start until Tuesday morning, the “booth crawl” was Monday evening – the odd ritual where swag-hungry attendees rush from booth to booth, grabbing all the free stuff they can carry, and having the occasional conversation with booth staff. Sometimes, the booth crawl can be a depressing experience, with people refusing to make eye contact, and just grabbing the stuff. But this was actually really great, with people excited about RDO, and wanting to learn more about we had to show.

Monday night was the Red Hat employee party at El Boo. It’s a boat-shaped restaurant sitting on one of the stone piers along the beach, and we had the whole place to ourselves for the entire night. It was a lot of fun. I stayed until midnight, when several friends toasted my birthday.

All together, it was a lovely start to the week. And there’s so much more to come.

RDO and CentOS

Continuing in the series about the RDO Meetup in Vancouver, in this recording we have Karsten Wade, of the CentOS project, talking about CentOS’s relationship with RDO, and with OpenStack in general. He talks about the CentOS build infrastructure, CI, package repos, and the CentOS Cloud SIG.

(If the player below doesn’t work for you, you can listen HERE.)

 

RDO on CentOS 7

With CentOS 7 now available, I quickly put it on my OpenStack demo laptop, and started installing RDO. It mostly just worked, but there were a few roadblocks to circumvent.

As usual, I followed the RDO Quickstart, so I won’t duplicate those steps here, in detail, but it goes like:


sudo yum update -y && sudo yum install -y http://rdo.fedorapeople.org/rdo-release.rpm && sudo yum install -y openstack-packstack && packstack --allinone

Comparison of string with 7 failed

The first problem occurs pretty quickly, in prescript.pp, with the following error message:

Comparison of String with 7 failed

This is due to the change in CentOS versioning scheme – the latest release of CentOS is version 7.0.1406, which is not a number. The script in question assumes that the version number is a number, and does a numerical comparison:


if $::operatingsystem in $el_releases and $::operatingsystemrelease < 7 {
...

This fails, because $::operatingsystemrelease is a string, not a number.

The solution here is to edit the file /usr/lib/python2.7/site-packages/packstack/puppet/templates/prescript.pp and replace the variable $::operatingsystemrelease with $::operatingsystemmajrelease around line 15.

While you’re at it, do this for every file in that directory, where $operatingsystemrelease is compared to 7.

See https://bugzilla.redhat.com/show_bug.cgi?id=1117035 for more detail, and to track when this is fixed.

mysql vs mariadb

The second problem, I’m not sure I understand just yet. The symptom is that mysql.pp fails with


Error: Could not enable mysqld:

To skip to the end of the story, this appears to be related to the switch from mysql to mariadb about a year ago, finally catching up with CentOS. The related bug is at https://bugzilla.redhat.com/show_bug.cgi?id=981116

The workaround that I used was:

# rm /usr/lib/systemd/system/mysqld.service 
# cp /usr/lib/systemd/system/mariadb.service /usr/lib/systemd/system/mysqld.service
# systemctl stop mariadb
# pkill mysql
# rm -f /var/lib/mysql/mysql.sock

Then run packstack again with the generated answer file from the last time.

However, elsewhere in the thread, we were assured that this shouldn’t be necessary, so YMMV. See https://www.redhat.com/archives/rdo-list/2014-July/msg00055.html for further discussion.

That’s all, folks

After those two workarounds, packstack completed successfully, and I have a working allinone install.

Hope this was helpful to someone.

UPDATE: The next time through, I encountered https://ask.openstack.org/en/question/35705/attempt-of-rdo-aio-install-icehouse-on-centos-7/

The workaround is to replace contents of /etc/redhat-release with “Fedora release 20 (Heisenbug)” and rerun packstack.

Turns out that this also fixes the mysql/mariadb problem above without having to go through the more complicated process.