Tag Archives: centos

CERN CentOS Dojo, part 4 of 4, Geneva

1 Comment

This is part 4 of a series about my visit to CERN in Geneva. You can read the entire series here: https://drbacchus.com/cern-centos-dojo-2017/

On Friday evening, I went downtown Geneva with several of my colleagues and various people that had attended the event.

CERN is right on the France/Switzerland border, so we’ve been going back and forth between the two countries several times a day, often not really knowing what country we were actually in.

I had been to Geneva when I was younger, but I really couldn’t say for sure when that was. The only thing I remember was the fountain – the Jet D’Eau – so I wanted to see that again. It was every bit as impressive as I remembered it.

CERN and Geneva

However, it was the end of a very long day, and between that, and jet lag, I was absolutely exhausted, so headed back to the hotel. I hope to go downtown again for a few hours this afternoon, but I kind of wanted to get these articles written while the memories were fresh.

When I was a kid, I dreamed that some day I would have a job traveling around the world, getting paid to see cool things. I think a lot of people dream of that. I have had the amazing good luck to achieve that goal. I have the best coworkers in the world, and I get to do things that I’m passionate about, every single day. The only way that this could be better is if I could have my beloved travel with me. Perhaps some day.

 

CERN CentOS Dojo, part 3 of 4: Friday Dojo

1 Comment

On Friday, I attended the CentOS Dojo at CERN, in Meyrin Switzerland.

CentOS dojos are small(ish) gatherings of CentOS enthusiasts that happen all over the world. Each one has a different focus depending on where it is held and the people that plan and attend it.

You can read more about dojos HERE.

On Friday, we had roughly 60-70 people in attendance, in a great auditorium provided by CERN. We had 97 people registered, and 75% is pretty standard turnout for free-to-register events, so we were very pleased.

You can get a general idea of the size of the crowd in this video:

The full schedule of talks can be seen here: https://indico.cern.ch/event/649159/timetable/#20171020

There was an emphasis on large-scale computing, since that’s what CERN does. And the day started with an overview of the CERN cloud computing cluster. Every time I attend this talk (and I’ve seen it perhaps 6 times now) the numbers are bigger and more impressive.

CERN and Geneva

This time, they reported 279 thousands cores in their cluster. That’s a lot. And it’s all running RDO. This makes me insanely proud to be a small part of that endeavor.

Other presentations included reports from various SIGs. SIGs are Special Interest Groups within CentOS. This is where the work is done to develop projects on top of CentOS, including packaging, testing, and promotion of those projects. You can read more about the SIGs here: https://wiki.centos.org/SpecialInterestGroup

If you want to see your project distributed in the CentOS distro, a SIG is the way to make this happen. Drop by the centos-devel mailing list to propose a SIG or join an existing one.

The entire day was recorded, so watch this space for the videos and slides from the various presentations.

The CERN folks appeared very pleased with the day, and stated their intention to do the event again on an annual basis, if all works out. These things aren’t free to produce, of course (even though we strive to make them always free to attend) so if your organization is interested in sponsoring future dojos, please contact me. I’ll also be publishing a blog post over on seven.centos.org in the coming days about what’s involved in doing one of these events, in case you’d like to host one at your own facility..

CERN Centos Dojo, event report: 2 of 4 – CERN tours

3 Comments

(This post is the second in a series of four. They are gathered here.)

The second half of Thursday was where we got to geek out and tour various parts of CERN.

I was a physics minor in college, many years ago, and had studied not just CERN, but many of the actual pieces of equipment we got to tour, so this was a great privilege.

We started by touring the data center where the data from all of the various physics experiments is crunched into useful information and discoveries. This was amazing for a number of reasons.

From the professional side, CERN is the largest installation of RDO – the project I work with at work – that we know of. 279 thousand cores running RDO OpenStack.

For those not part of my geek world, that translates into hundreds of thousands of physical computers, arranged in racks, crunching data to unlock the secrets of the universe.

For those that are part of my geek world, you can understand why this was an exciting thing to see in person and walk through.

The full photo album is here, but I want to particularly show a couple of shots:

Visiting CERN

Here we have several members of the RDO and CentOS team standing in front of some of the systems that run RDO.

Visiting CERN

And here we have a photo that only a geek can love – this is the actual computer on which the very first website ran. Yes, boys and girls, that’s Tim Berners-Lee’s desktop computer from the very first days of the World Wide Web. It’s ok to be jealous.

There will also be some video over on my YouTube channel, but I haven’t yet had an opportunity to edit and post that stuff.

Next, we visited the exhibit about the Superconducting Super Collider, also known as the Large Hadron Collider. This was stuff that I studied in college, and have geeked out about for the years since then.

There are pictures from this in the larger album, but I want to point out one particular picture of something that absolutely blew my mind.

Most of the experiments in the LHC involve accelerating sub-atomic particles (mostly protons) to very high speeds – very close to the speed of light – and then crashing them into something. When this happens, bits of it fly off in random directions, and the equipment has to detect those bits and learn things about them – their mass, speed, momentum, and so on.

In the early days, one of the the ways that they did this was to build a large chamber and string very fine wires across it, so that when the particles hit those wires it would cause electrical impulses.

Those electrical impulses were captured by these:

CERN visit

Those are individual circuit boards. THOUSANDS of them, each individually hand-soldered. Those are individual resistors, capacitors, and ICs, individually soldered to boards. The amount of work involved – the dedication, time, and attention to detail – is simply staggering. This photo is perhaps 1/1000th of the total number of boards. If you’ve done any hand-soldering or electronic projects, you’ll have a small sense of the scale of this thing. I was absolutely staggered by this device.

Outside on the lawn were several pieces of gigantic equipment that were used in the very early days of particle physics, and this was like having the pages of my college text book there in front of me. I think my colleagues thought I’d lost my mind a little.

College was a long time ago, and most of the stuff I learned has gone away, but I still have the sense of awe of it all. That an idea (let’s smash protons together!) resulted in this stuff – and more than 10,000 people working in one place to make it happen, is really a testament to the power of the human mind. I know some of my colleagues were bored by it all, but I am still reeling a little from being there, and seeing and touching these things. I am so grateful to Tim Bell and Thomas Oulevey for making this astonishing opportunity available to me.

Finally, we visited the ATLAS experiment, where they have turned the control room into a fish tank where you can watch the scientists at work.

CERN visit

What struck me particularly here was that most of the people in the room were so young. I hope they have a sense of the amazing opportunity that they have here. I expect that a lot of these kids will go on to change the world in ways that we haven’t even thought of yet. I am immensely jealous of them.

So, that was the geek chapter of our visit. Please read the rest of the series for the whole story.

CERN Centos Dojo 2017, Event report (1 of 4): Thursday meeting

1 Comment

Over the last few days I’ve been in Geneva for the CERN CentOS Dojo, 2017 edition.

(This is part 1 of a series of four posts. They are gathered here.)

On Thursday, prior to the main event, a smaller group of CentOS core community got together for some deep-dive discussions around the coming challenges that the project is facing, and constructive ways to address them.

This meeting was very potentially productive. I say potentially because some great decisions were made, with universal approval, but everything depends on the execution. Some of these decisions will take a great deal of work over the coming months. Of course, nobody is averse to hard work, but we all also have other things to do. So we need to keep the long-term health of the project firmly in mind, and find time for these tasks.

 

The full notes from that meeting have been posted to the Centos-devel mailing list for further discussion.

The attendees were from many different organizations, countries, and cultures. While the various organizations represented have rather different goals and motivations, there was great unity of purpose – ensuring the long-term health of the CentOS project.

Topics covered were focused on removing roadblocks to forward movement on the project, and removing obstacles to new contributors to the project coming on board and getting things done. This was very encouraging.

We were disappointed that a number of prominent community members were unable to attend. Notably, Karanbir was absent due to a broken toe:

 

Continuing discussion of the topics will happen on the centos-devel mailing list, and, as always, people who want to step up to assist in any of the identified tasks are encouraged to speak up and volunteer.

CentOS is a community of project communities, and works best when those projects identify the things that will make them more productive, and then step up to make those things happen.

 

CERN Centos Dojo 2017, Event report (0 of 4)

3 Comments

For the last few days I’ve been in Geneva for the CentOS dojo at CERN.

What’s CERN? – http://cern.ch/

What’s a dojo? – https://wiki.centos.org/Events/Dojo/

What’s CentOS? – http://centos.org/

A lot has happened that I want to write about, so I’ll be breaking this into several posts:

(As usual, if you’re attempting to follow along on Facebook, you’ll be missing all of the photos and videos, so you’ll really want to go directly to my blog, at https://drbacchus.com/)

 

Student Cluster Competition

Last week, as I mentioned in my earlier post, I was in Frankfurt, Germany, for the ISC High Performance Computing conference. The thing that grabbed my attention, more than anything else, was the Student Cluster Competition.  11 teams from around the world – mostly from Universities – were competing to create the fastest (by a variety of measures) student supercomputer. These students have progressed from earlier regional competitions, and are the world’s finest young HPC experts. Just being there was an amazing accomplishment. And these young people were obviously thrilled to be there.

Each team had hardware that had been sponsored by major HPC vendors. I talked with several of the teams about this. The UPC Thunderchip team, from Barcelona Tech, (Winner of the Fan Favorite award!) said that their hardware, for example, had been donated by (among other vendors) CoolIT systems, who had donated the liquid cooling system that sat atop their rack.

(When I was in college, we had a retired 3B2 that someone had dumpster-dived for us, but I’m not bitter.)

Over the course of the week, these teams were given a variety of data challenges. Some of them, they knew ahead of time and had optimized for. Others were surprise challenges, which they had to optimize for on the fly.

While the jobs were running, the students roamed the show floor, talking with vendors, and, I’m sure, making contacts that will be beneficial in their future careers.

Now, granted, I had a bit of a ulterior motive. I was trying to find out the role that CentOS plays in this space. And, as I mentioned in my earlier post, 8 of the 11 teams were running CentOS. (One – University of Hamburg – was running Fedora. Two – NorthEast/Purdue, and Barcelona Tech – were running Ubuntu) And teams that placed first, second, and third in the competition – (First place: Tsinghua University, Beijing. Second place: Centre for High Performance Computing South Africa. Third place: Beihang University, Beijing.) – were also running CentOS. And many of the research organizations I talked to were also running CentOS on their HPC clusters.

I ended up doing interviews with just two of the teams, about their hardware, and what tests that they had to complete on them to win the contest.

At the end, while just three teams walked away with the trophies, all of these students had an amazing opportunity. I was so impressed with their professionalism, as well as their brilliance.

And good luck to the teams who have been invited to the upcoming competition in Denver. I hope I’ll be able to observe that one, too!

Things I’ve learned at ISC HPC

1 Comment

I came into the ISC event pretty ignorant. Here’s some of the things I’ve learned.

Supercomputers run Linux. All of them. This isn’t even a topic of discussion. Yes, I’m sure there are some that don’t, but everyone here just assumes that you are running Linux. And probably two or three Apache products.

Supercomputing isn’t about software. This is a hardware conference.

Supercomputing is primarily about how fast you can get rid of heat. And these people are serious about cooling. I’ve seen some amazingly cool cooling rigs. Perhaps the coolest of them was this one: https://youtu.be/hs9WG0ZA79Q  That unit is called the AIC24, and is manufactured by Asperitas, and is a full submersion rack. You lower your blades into oil, which is in turn cooled by a water cooling pump. This is much quieter than fans, and much more efficient. The oil was cool enough to touch. Enormous supercomputing centers are locating on the edge of lakes specifically so that they can pump cool water from the lake into cooling units like this.

I also saw this cool demo: https://youtu.be/aaEQN8DH0kM  You can actually see the oil boiling on the processor. The vapor is then condensed on a cooling unit in the back and trickles back down into the tank.

I have also been blown away by the Student Cluster Competition. These kids have access to hardware that would have blown my mind when I was in school. There’s 11 teams competing on a variety of metrics, and they have these astonishing supercomputers at their disposal. I was also amazed to discover that LINPACK is still one of the standard benchmarks. I used that when I was in college!

The student hardware is all sponsored by the vendors that are here at this event – presumably so that they can benefit from the publicity when they win the contest. Check out some of these rigs:

I was pleasantly pleased to discover that of the 11 teams competing, 8 are running CentOS. One other was running Fedora – they wanted to run CentOS, but needed a newer kernel for something (I wasn’t very clear on what that was. I’ll try to go find out more information today.) The other two were running Ubuntu. CentOS  also appears to be the preferred platform for the various research institutes I’ve talked to. However, these are the groups that chose to come over to the Red Hat booth and talk to me, so I do acknowledge that this is a rather self-selected sample. The sign on the SuSE booth claims that SuSE is the Linux “most used by the top 100 supercomputers.” More research is warranted here. But it appears clear, at least from this small sample, and from conversations with the students, that CentOS is just What You Run when you’re doing supercomputing.

And finally, I’ve learned (not that it’s a big surprise) that one year of high school German, 30 years ago, is not a great deal of help. And that people are amazingly patient and kind with my ignorance – something that I’ve discovered almost everywhere in the world

 

Moving to CentOS

TL;DR: Leaving OpenStack; Moving to CentOS; Still at Red Hat.

4 years ago, I came to Red Hat, and started as the OpenStack Community Liaison, working primarily with the RDO project, but more generally with all of Red Hat’s involvement in the upstream OpenStack project.

I took over from Dave Neary, but it took a while to actually replace him. His depth of knowledge and experience with the community were not easy to step into.

Over those 4 years, I’ve become much more knowledgeable about OpenStack – the community as well as the technology. It’s a wonderful community, with a passion for open source, for doing things transparently and collaboratively, and for doing things well. The individuals in the community have been great to get to know – both people here at Red Hat, as well as people in other organizations, and at the OpenStack Foundation. I could certainly call out dozens of individuals who have made my time with OpenStack smoother. The names that come to mind are Haikel Guemar, Stefano Maffuli, Perry Meyers, Eliska Malikova, Alan Pevec, Jakub Ruzicka, Rain Leander … see, I knew that as soon as I got started I would find that there’s no end in sight.

Rain, in particular, has really stood out as someone that was hugely passionate about the community around OpenStack, and was just such a delight to work with, particularly when we were able to attend the same events and work together in person.

This is why I’m so excited to announce that Rain will be taking my place as the RDO/OpenStack community manager for Red Hat, effective immediately. I cannot think of anybody more qualified, in skills and temperament, for this position, and I am completely confident that I’m leaving the community in good hands. One develops a lot of ownership of a project over four years, but I have no doubt she’ll take care of the project.

I’m not leaving Red Hat, though. Instead, I’m moving over to be more active in the CentOS community. CentOS is an exciting community that is absolutely critical to Red Hat. It’s the place where community projects, like RDO, as well as many others, do their development and testing, before being deployed and supported on Red Hat Enterprise Linux. I’ll be focusing on a variety of things, including CentOS in HPC (High Performance Computing) and IoT (Internet of Things).

CentOS presents a number of challenges from a community perspective, and I’m very pleased to be more active there. It will be an interesting and challenging place to be, and I’ll be, once again, working with an awesome group of people. I’m sure I’ll be telling lots of CentOS stories on this website in the years to come, so stay tuned.

Follow RDO on Twitter at @rdocommunity and follow CentOS on Twitter at @centos to keep up to date with the two communities, as well as learn how we work together.

RDO and CentOS

1 Comment

Continuing in the series about the RDO Meetup in Vancouver, in this recording we have Karsten Wade, of the CentOS project, talking about CentOS’s relationship with RDO, and with OpenStack in general. He talks about the CentOS build infrastructure, CI, package repos, and the CentOS Cloud SIG.

(If the player below doesn’t work for you, you can listen HERE.)

 

RDO on CentOS 7

With CentOS 7 now available, I quickly put it on my OpenStack demo laptop, and started installing RDO. It mostly just worked, but there were a few roadblocks to circumvent.

As usual, I followed the RDO Quickstart, so I won’t duplicate those steps here, in detail, but it goes like:


sudo yum update -y && sudo yum install -y http://rdo.fedorapeople.org/rdo-release.rpm && sudo yum install -y openstack-packstack && packstack --allinone

Comparison of string with 7 failed

The first problem occurs pretty quickly, in prescript.pp, with the following error message:

Comparison of String with 7 failed

This is due to the change in CentOS versioning scheme – the latest release of CentOS is version 7.0.1406, which is not a number. The script in question assumes that the version number is a number, and does a numerical comparison:


if $::operatingsystem in $el_releases and $::operatingsystemrelease < 7 {
...

This fails, because $::operatingsystemrelease is a string, not a number.

The solution here is to edit the file /usr/lib/python2.7/site-packages/packstack/puppet/templates/prescript.pp and replace the variable $::operatingsystemrelease with $::operatingsystemmajrelease around line 15.

While you’re at it, do this for every file in that directory, where $operatingsystemrelease is compared to 7.

See https://bugzilla.redhat.com/show_bug.cgi?id=1117035 for more detail, and to track when this is fixed.

mysql vs mariadb

The second problem, I’m not sure I understand just yet. The symptom is that mysql.pp fails with


Error: Could not enable mysqld:

To skip to the end of the story, this appears to be related to the switch from mysql to mariadb about a year ago, finally catching up with CentOS. The related bug is at https://bugzilla.redhat.com/show_bug.cgi?id=981116

The workaround that I used was:

# rm /usr/lib/systemd/system/mysqld.service 
# cp /usr/lib/systemd/system/mariadb.service /usr/lib/systemd/system/mysqld.service
# systemctl stop mariadb
# pkill mysql
# rm -f /var/lib/mysql/mysql.sock

Then run packstack again with the generated answer file from the last time.

However, elsewhere in the thread, we were assured that this shouldn’t be necessary, so YMMV. See https://www.redhat.com/archives/rdo-list/2014-July/msg00055.html for further discussion.

That’s all, folks

After those two workarounds, packstack completed successfully, and I have a working allinone install.

Hope this was helpful to someone.

UPDATE: The next time through, I encountered https://ask.openstack.org/en/question/35705/attempt-of-rdo-aio-install-icehouse-on-centos-7/

The workaround is to replace contents of /etc/redhat-release with “Fedora release 20 (Heisenbug)” and rerun packstack.

Turns out that this also fixes the mysql/mariadb problem above without having to go through the more complicated process.