Supply Chain Attacks
Supply chain attacks are extremely common but not always quick to be detected. This morning I saw an article about the AUR (Arch User Repository), a community-driven repository created and managed by Arch Linux users, hosting malicious software. The software in question was an orphaned PDF Viewer called “acroread”. This software would collect mess with systemd, collect system information and exfiltrate that data . For more information on that please read the hacker news article referenced.
The Supply Chain Problem
A supply chain is actually a complex and dynamic supply and demand network.
A supply chain is a system of organizations, people, activities, information, and resources involved in moving a product or service from supplier to customer.  A supply chain attack when when you exploit part of the process to change the end result.
I am guessing it’s because of the Gentoo  and Docker hub  attacks that everyone is looking at their community packages and we will see more of these reports coming out. With bitjacking there is a monetary incentive to hack a “trusted”, but in reality community, supply chain. This is the dark side of the open code movement.
Supply chain management and transparency is hard. Double that if you are a community supported entity that doesn’t make money or give guarantees. Do you spend time fixing bugs, adding features, reviewing pull requests, documenting old features better, document new features, demand/community generation, and so many more actions. When there is not a level of control but an implied trust… you are asking for problems. Ask yourself, would you run a binary from a random guy on the internet? If so, look inward.
I want to point that that this was discovered in the AUR (Arch User Repository), a community-driven repository created and managed by Arch Linux users, not official repositories like Arch Build System (ABS).
We think about supply chain management a lot at Red Hat. I would talk about this often when I was a Solution Architect. Red Hat, Fedora, Suse and Ubuntu deliver each have repositories that are controlled and backed by companies. This means they have control to the kingdom and you need to follow a specific process to get added. The AUR appears to have something similar but since it’s community based, they found a weakness in an orphaned package and use that as a jumping on point.
Don’t run some code a guy on the internet wrote… Get it from a trusted source. I’m looking at you snaps and juju… Unfortunately, scammers and state actors understand you want things ‘easy’. There is a contention between ease of use and security. Moving fast and moving safely. Over and over again we see systems get abused. Be mindful of your sources and hopefully Arch and Gentoo doesn’t take to much of a reputation hit from this, but hopefully they are more vigilant in the future.
OpenStack Summit – 2018 – Vancouver
Attendance this year in Vancouver was way down, about 3,500, from Boston last year. The energy level of the keynotes, sessions and hallway were muted. The expo hall was also much smaller than I have ever seen in my past 3 years attending the NA Summits.
The general focus of the OpenStack Foundation is on inclusion and moving up the stack. They are seeing what developers and cloud native customers are talking about to show how it runs on OpenStack. It feels genuine they are trying to help, but with the attention being so much on Kubernetes, OpenStack capabilities kept getting upstaged.
I’ve been in the OpenStack orbit for about 6 years now, starting with Essex. I started as a consultant building OpenStack clouds for development. I’ve helped organizations move smaller private clouds to production. Most recently I was helping customers adopt OpenStack. I want to see OpenStack succeed. I would love to see more/any hybrid cloud solutions and integrations points with other environments. How great would it be for Horizon to connect into your AWS or Azure console? What if Horizon integrated with Kubernetes dashboard? Instead, OpenStack Foundation says, let’s talk about multiple container solutions…
The general shrinking of attendance I don’t see necessarily as a sign of the end. I think there are many factors that caused this to nose dive:
- Convention fatigue
- May 2 – 4. Kubecon. Copenhagen, Denmark. Mass popularity in public and private cloud. International travel.
- May 8 – 10. Red Hat Summit. San Francisco, CA. Broader Open Source and Hybrid Cloud Focus.
- May 7 – 9. Microsoft Build. Seattle, WA. Broad industry topics and leading cloud provider.
- May 9 – 17. Pycon. Cleveland, OH. Python being the primary language OpenStack is written in, and the largest Python event in North America.
- May 8 – 10. Google I/O. Mountain View, CA.
- May 21 – 24. OpenStack Summit. Vancouver, BC. The potential attendees and sponsors of this event in the month of May alone probably wanted to attend multiple of the above events. The overlap is huge. Which would you rather attend, have a booth or present?
- Twice a year. There comes a time when your project isn’t the most innovative and changing beast in the enterprise, this is a good thing. Does OpenStack still need to release two versions a year? Does it need to have two conventions in a year? I’m not one to say but I’d love to hear what you think…
- International Travel. I live in the United States and traveling to Canada is not much of a problem, but it can be for some. If your company is not world wide or have a presence in Canada I have found there is a LOT of hesitation about sending your employees to any other country. Each company is going to put a different weight to this, but it is part of the equation. Also, Canada requires US Citizens to have a Passport, and that’s surprisingly uncommon for people to have in IT. You can’t just go to Canada on a whim and the Passport process can take 6 – 8 weeks or more.
- Vancouver, again. This is another minor point but something they should consider. Should we go somewhere new or go to a classic spot? It seems like in NA they are hitting the old haunts. Maybe, you should go to places where it’s not “Just another convention” like Chicago or Detroit. With it being centrally located it allows people coming from all over to fly for 2 – 3 hours instead of 5 – 8 hours. I got to San Francisco and Las Vegas to many times a year for conventions. Let’s try somewhere different.
I want to see OpenStack continue to succeed. They’ve made some missteps in the past trying to do new things. Kubernetes is cashing the immutable application check OpenStack wrote 6 years ago. The product is maturing and the hype is drying. However, I hope I’ve made clear that I think it’s not at all dead. It’s in an important phase of it’s life. Middle aged. Accept it. A fancy new convertible and black hair dye won’t change the fact you’ve grown up. That’s a good thing.
Videos to watch
Is the public cloud really eating OpenStack’s lunch?
OpenStack upgrade strategy the fast forward upgrade
Root your OpenStack on a solid foundation of leaf spine architecture
Integrating keystone with large scale centralized authentication
Leveraging serverless functions in heat to deliver anything as a service-xaas
The illusion of infinite capacity part 2 – over subscription
Free software needs free tools
Workload optimized OpenStack made easy
Engineering container security
Container Linux and RHEL: The road ahead
This is from the Red Hat Summit 2018 series.
I’m not the person to give you the best updated road map for RHEL and especially Container Linux. I’ve been focused in the multi-cloud Management and automation world for the past few years. A lot of developments have been afoot. Here are some amazing highlights with my commentary. I strongly suggest that you check it out the video below.
Immutable Foundation for OpenShift
The new Red Hat CoreOS (RHCOS) focus on several key areas. Red Hat Enterprise Linux and RHCOS are designed for different use cases and it’s important not to mix them. RHCOS will keep focused on one major thing… OpenShift/Kubernetes. RHCOS will be on the same cadence as Red Hat OpenShift Container Platform (RHOCP). This means when Kubenetes and OpenShift release, RHCOS will release. This allows engineering to keep the same speed and make changes necessary for new release of RHOSP. This also means that when we sunset specific version of RHOSP, it will also sunset the locked in version of RHCOS. Now we can keep the release coupled and supportable without having to drag RHEL before it’s ready.
There are so many more changes, but I don’t just want to copy slides. Hopefully you are intrigued enough to watch the video. Thanks for reading!
Red Hat Summit Recap
I spent last week in San Francisco attending Red Hat Summit 2018. This is my third Red Hat Summit in a row. The experience is always a blur and a whirlwind of activity from beginning to end. I am going to try and put together a few post this week of things I thought significant to the direction of Red Hat and the industry.
I had an amazing time meeting with friends and colleagues. I spent quite a bit of time helping customers at the Public Cloud booth. The booth was suppose to help customers understand the portability of their subscriptions to the public cloud via the Cloud Access Program but most of the focus for customers was running OpenShift on the public cloud.
A recap of Red Hat Summit 2018.
OpenShift with CockroachDB
“Red Hat® OpenShift is a container application platform that brings Docker and Kubernetes to the enterprise. Regardless of your applications architecture, OpenShift lets you easily and quickly build, develop, and deploy in nearly any infrastructure, public or private. ” 
I’ve helped a lot of customers find their way to OpenShift. I’ve helped them develop and refine use case as well as figure out how it fits into their environment.
Other than having an unfortunate name I really like Cockroach Labs CockroachDB from the little I’ve used it. I am by far not an expert on CockroachDB. I first learned about it at OpenStack Summit 17 in Boston. Kudo to the guys for putting this presentation together and presenting it on the big stage.
What is CockroachDB?
“CockroachDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. It scales horizontally; survives disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention; supports strongly-consistent ACID transactions; and provides a familiar SQL API for structuring, manipulating, and querying data.” 
There is a lot more to be said about the statement above that I am not going to cover here. To summarize, you can partition and replicate your database while still making sure queries are only getting the latest data, called “Strong Consistancy”.
On a personal level I like the notion of OpenShift and CockroachDB together for a couple reasons. Chiefly, they are both Open Source. You can experiment locally with no up front cost on a developer macine and eventually roll it into production. When you are ready for production both projects have Enterprise support offerings. It’s not critical you invest time, money and resources into figuring out if it’s going to work for you just to find out 6-9 months down the line it’s not what you hoped it would be. Experiment now! My next post will give you the framework to start….
New year for Wizard
2018 is upon us and I’ve renewed my interest in posting to this blog. I’ve been doing a ton of learning in 2017 and I think 2018 is going to be the year I start spreading that knowledge.
I’m always pushing myself to look for new ground or enhancing new concepts that we are familiar with already. Look for more of this in coming weeks. I think I have things to say about data storage (Databases) in the kubernetes space.
Intro to CloudForms Tags
This blog post was originally posted by myself on Blogger (8/26/2016).
Red Hat CloudForms offers unified management for hybrid environments, providing a consistent experience and functionality across container based infrastructures, virtualization, private and public cloud platforms. Red Hat CloudForms enables enterprises to accelerate service delivery by providing self service, including complete operational and lifecycle management of the deployed services. It provides greater operational visibility through continuous discovery, monitoring and deep inspection of managed resources. And it ensures compliance and governance via automated policy enforcement and remediation. All the while, CloudForms is reducing operational costs, reducing or eliminating the manual processes that burden IT staff.
For more information visit http://redhat.com/cloudforms
I think tags are one of the most important features of Red Hat CloudForms. CloudForms ability to tag resources for later use in reporting, chargeback/showback and automation is critical for getting more in-depth knowledge and generating laser focused reports that provide value.
In this article I am going to touch on general guidelines I use when building a tag schema. I believe there are two rules when talking about CloudForms tags. It’s better too over tag your resources than under tag. If you can measure it; you can manage and monitor it. Just like any data structure, a well thought out schema will save you a lot of work.
Tag Schema Recommendations
The most important thing about your business tag schema is they make sense to you and your companies. The examples I list below are a very rough estimation of what your business will look like and how it will operate. Think about logically grouped business resources and come up with a tag for them.
– IT Development
– IT Operations
IT Operations Tags
If you’re reading this blog post, these tags probably matter most to you. Remember, measure what matters most to you. Help the business understand your value and what you do. I know it, you hopefully know it, let them know it too.
– Production Systems
– Development Lab
– Solid State Hard drive (SSD)
– Dell Hardware
– PostgreSQL 5
– Oracle DB 11G
– Web Server
– New York Datacenter
– Hong Kong Datacenter
– Zone A – Virginia DC
– Zone B – Virginia DC
Imagine if IT and the business came to an agreement on service windows based on what worked for each business unit. This can happen. Maybe you want to have test deployments on production resources. Tagging change windows into your resources will help with reporting and also automation.
– Patch Window A (Second Tuesday of the month)
– Patch Window B (Last Sunday of the month)
– Canary Deployment Environment
A service level agreement (SLA) is a contract between a service provider (either internal or external) and the end user that defines the level of service expected from the service provider.
Security tagging is something I’m still working through. I know there is value in creating a Security Role, Group and Users for Dashboard and Reporting. I’m looking at linking this into Policies and exporting log events to a SIM (Security Information Management).
– Information Assurance Security Team Check
– IA Validated
– IA Not Checked
– Service Catalog Provisioned (Provisioned Machines Certified Gold Master)
– Satellite Verified