By Tim Boles | Edited by Jim Czuprynski 

Over the coming weeks, SELECT and DBTA will be collaborating on a series of articles that have DevOps as their core theme. Like so many other buzz words in Information Technology today, there is no single agreed upon definition that has been set forth for DevOps, so the editors at SELECT decided that before we publish articles on DevOps, we should first describe what we view DevOps really means.

The Elusive Definition of DevOps

A simple search of the Internet for the term DevOps reveals many different definitions:

DevOps is a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality. (Bass, Len; Weber, Ingo; Zhu, Liming. DevOps: A Software Architect’s Perspective)

DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. (https://www.gartner.com)

DevOps (development and operations) is an enterprise software development phrase used to mean a type of agile relationship between development and IT operations. (https://www.webopedia.com)

DevOps is an approach to lean and agile software delivery that promotes closer collaboration between lines of business, development and IT operations. (https://www.ibm.com)

It is hard to just use one or two sentences to convey the meaning of DevOps. Most of the pages listed describe in great detail the methodology and whys, hows, and hoped-for results of DevOps. However, it’s important to note the similarities in the four definitions above: They all use words and phrases like “reduce the time,” “rapid IT service delivery,” and “agile.” One of the central themes of DevOps is thus obvious: it is an attempt to develop software applications speedily to a production-ready mode significantly faster than the traditional waterfall software development lifecycle models of the past.

The other obvious focus of DevOps is the significant cultural shift within an IT organization. Instead of the historical business, architecture, development, testing, and operations silo approach, there is an emphasis on collaboration and a closer meshing together of these teams. That’s probably the most significant reason for the moniker DevOps – it’s a mashup of the terms Development and Operations.

CRISP and CALMS: DevOps Core Values

Aniket Mhala recently commented in The Agile Admin, mentioning the CRISP approach to DevOps, and I think it is right on target. The CRISP approach to DevOps reflects what can be thought of as the core values of DevOps.

Here’s a brief summary of the CRISP approach:

C: Collaboration between Dev and Ops (One Team – One Goal)

R: Repeatable process that delivers a high-quality, stable application

I: An Iterative and Incremental approach for continuous improvement

S: A Saleable process across multiple stakeholders and teams in the organization

P: A Process-oriented, collaborative approach centered on People, Principles and Practices

This definition is general enough to highlight the approach but does not lean on any particular set of tools or methods.

Damon Edwards and John Willis at DevOpsDays Mountainview coined the acronym CAMS – Culture, Automation, Measurement, and Sharing – to describe its core values. Later, Jez Humble, the co-author of “The DevOps Handbook.” added an L to the CAMS acronym: Culture, Automation, Lean, Measurement and Sharing. Let’s take a detailed look at what the CALMS definition brings to light about DevOps.

Culture

It will probably not come as a shock to those of us with even a few years of experience in IT that in many organizations there is adversity between groups within the organization. Operations administrators may refuse to allow developers any type of access to production systems; developers feel that their responsibility ends with delivery of code to the production environment; testers and quality assurance managers may believe that unit testing should be completed by developers who often believe it is not their job to test. Within some organizations, asking for help from people outside one’s core group might even be seen as an admission of a serious lack of skills, or even incompetence.

One of the core values of DevOps is a culture of shared responsibility, and that means the attitude of “It’s done – throw it over the wall” is simply not permitted. Everyone on the team is involved with the success or failure of system changes. Yet shared responsibility is not just about teamwork – it is also about breaking down barriers between teams and getting disparate groups of people to work together as one unit.

Automation

The more human intervention is needed in a process, the more likely there are to be errors during the process. My personal experience with delivering application software was that immediately after it was delivered, our team would spend a significant part of a weekend with installation, system testing, and then user acceptance testing.

The good news is that as technology matures and advances there has been additional opportunities to automate. One of the core values of DevOps therefore focuses on looking for ways to automate not only the process of moving changes into production, but the entire system lifecycle. Automation can be integrated throughout the process of software development, including release management, configuration management, testing, monitoring and control, provisioning, and system integration. Gains in productivity are achieved with automation by reducing human errors, reducing defects, creating consistency, and even enabling self-service.

Lean

The concept of lean is to continuously improve the ability to deliver a quality product by lessening or eliminating wasted time and effort. Some of the areas of focus should be:

Flow. Activities should be performed with minimal interruptions and via automation whenever possible.

Minimizing interruptions can be accomplished in many ways. In our group we constantly work to remove non-value automated messages, reduce false alerting, and automate routine tasks. The fewer interruptions that occur from these types of items translates to more time that a DBA can spend focused on DevOps activities. A prime example of this in our environment was to automate the process of testing our database restores every month. This activity was routine that would take 20 minutes or more of a team member’s time to complete per database, and with well over 100 databases to test. Now it is done automatically each night, and we get a report of all successes and failures. This meant our team reclaimed nearly a full person-week of time every month, or almost three person-months of previously wasted time per year.

Pull. Provide what the customer wants when they want it, focusing on quick turnaround.

I think of this as being ready to move quickly and decisively, utilizing skillsets from various groups. A prime example of this in my personal experience is the need a client had to find a new solution to their Business Continuity Volume (BCV) approaches. Much of their business, development and production processes depended on the BCVs. Various business units, development teams, infrastructure administrators, and DBAs collaborated to find precisely the right mixture of technologies and strategies that would meet everyone’s requirements. The new solution was in place well before the BCVs had to be discontinued.

Perfection. Concentrate on doing things right the first time, thus reducing errors.

Error reduction can be achieved by a variety of methods. One success story that I have involves the evolution of the change control process at a client site. System changes involved utilizing a long list of instructions written by developers. The instructions generally included list of files to download from a repository, copying the files to various directories on remote servers, and then executing scripts and SQL statements. We found that the larger the code based that was promoted, the longer it would take and the more possibility of needless human error. Our client implemented change control software from a Software Automation Solutions for Oracle EBS vendor. The developer instructions are still provided within Word documents as a cross-check, but 90% of the time it is eliminated by providing a file name to a GUI based program which is responsible for the remaining promotion tasks.

Measurement

The only way to really know if your DevOps approach is working is by measuring its success. Of course this begs the question of what should be measured, and using which methods of measurement?

As with pretty much everything else in DevOps, this is going to vary by organization in how it is implemented. It also depends on what the business organization that your team supports is most interested in measuring. From a purely technical perspective, this could mean measuring deployment frequency, deployment time, and even lead time – i.e. how long it takes to implement successfully a solution for particular requirements. Other useful metrics include problem tickets generated, average ticket resolution time, defect rate, application availability (a.k.a uptime), application performance, change volume, database response time, percentage of transaction time spent in database, resource utilization, and database query times. The concept is that the entire organization has a stake in DevOps, so each group will want to measure other areas to know if DevOps is working for them.

Sharing

Finally, this is the key part of the success of implementing DevOps at any organization. Often the focus is on developers and operation teams and creating a culture where people share ideas, pain points, and problems. However, when an issue arises, the DevOps approach avoids finger pointing and instead focuses the organization’s talent on solving the problem at hand; therefore, it goes beyond just developers and operation teams, and it should include the entire organization.

I’ve personally encountered on multiple occasions that the insight that helped solve a problem originated from a non-technical person within our own IT organization, or even from outside the technical people within our organization – sometimes even from an application user or business unit leader. Ideas and processes that can help an organization constantly improve often come from participating in user groups, conferences and in just sharing DevOps success with others.

DevOps and the DBA

So if you are an Oracle DBA, you’re probably wondering how your role will fit into an IT organization that’s decided to leverage the DevOps model for application development. The good news is that the core values of DevOps and the role of DBA are actually complementary instead of competitive or combative, especially when either the CRISP or CALMS approach to DevOps is put in place.

First, in the right culture, the role of the DBA is an intricate part of the software development lifecycle. Based on my experience as a DBA, you have already probably been in a similar situation: A call comes in from a trusted user complaining about a noticeable decrease in application performance after a code release, and they want to know if there is something wrong with the database. Once your heart stops pounding, you put on your DBA hat and begin the multi-vendor finger pointing exercise because you don’t necessarily know all of the details about what application changes might be affecting the database. Wouldn’t it simply be so much better if the DBA already knew where to start her research by already knowing exactly which changes had been implemented as part of the new release?

The DevOps DBA: A CALMS Approach

Let’s take a look at how the CALMS DevOps model would work for a DevOps DBA:

Collaboration. In the DevOps model, this intrinsic understanding of the newly deployed application comes naturally from collaboration across all departments. There should be DBA presence in meetings of every department that has a stake in the system architecture or software deployment; DBAs should be actively participating in the requirements review, architecture development, coding reviews, test meetings and operations weekly reviews. This way, the DBA team will know well in advance exactly what to expect after deployment of a new release – or at least where to start looking for issues, and (just as importantly!) where not to look as a part of the solution.

Automation. In the past a DBA proved her “chops” by using SQL*Plus and scripts to do all her database management work manually. But the DevOps DBA knows that doing things manually will almost certainly introduce the possibility of human error – through commission or omission! – and thus actually tends to contribute to the biggest reason for loss of time in any operational environment. The DevOps DBA therefore needs to focus on getting the right tools in place and automating repeatable tasks. The primary goal is to make the work of deploying application changes in a highly repeatable fashion, increasing the consistency and reliability of the DBA’s work.

Lean. When a DevOps DBA leverages automation for repetitive tasks, he naturally accomplishes the work in a faster time frame, and that starts in motion the tendency towards lean processing within the DevOps organization. And the time saved through these efficiency improvements eventually translate into more human resources that the DevOps DBA can transfer into ever-increasing collaboration and finding ways to make other existing processes even more lean.

Measuring. How does the DevOps DBA truly know if he’s make a process leaner? Simple – by measuring key performance indicators for the application development process itself. So what should a DevOps DBA be measuring and tracking? For starters, an excellent overall metric is the amount of time it used to take to deploy application releases manually versus using an automated process. As automation increases, the dual assumptions of faster deployments with fewer unexpected failures post-release should be confirmed. The DevOps DBA should also monitor and track crucial database performance metrics such as overall response time, average percentage of transaction time spent within the database, resource utilization (memory, CPU, and physical I/O), and database query times.

Sharing. Finally, the DevOps DBA isn’t afraid to share the results she’s measuring with her application developer teams, IT management, and even the business units the combined DevOps team supports – regardless of success or failure. But a DevOps DBA goes beyond just sharing pertinent metrics and what they mean; it also involves sharing her knowledge and expertise with her entire team.

Of all the DevOps core values, this one is most likely to be completely foreign to IT organizations whose teams have been “siloed” in the past to the point that even if developers are good at writing code and building software, they are typically ignorant of operational needs. Of course, the opposite is true as well; operational administrators are often unaware of the problems and challenges of application developers. Having an open conversation about problems, challenges and needs will often bring forth solutions that would otherwise never be heard or even considered.

The DevOps DBA: A CRISP Approach

How would a DevOps DBA leverage the core values of the CRISP model in his role on a DevOps-oriented team? It will vary in detail depending on the organization and technology being used:

• Collaboration can be face-to-face or electronic like using Skype or similar messaging tools
• Processes can made repeatable by favorite tools and languages like Windows Powershell, PL/SQL, JavaScript, PHP, Oracle Scheduler or shell scripts just to name a few. Responses to potential problems can be addressed using Oracle Enterprise Manager Cloud Control incident management. You could also use tools like Oracle Change Management to create scripts to apply against environments to synchronize them.
• DBAs are generally very good at implementing small changes to a stable application system over time, so the iterative and incremental approach is right up their alley.
• The saleable, process-oriented collaborative approach go hand in hand in that the DevOps DBA will often have to work closely with groups and educate them to get their buy off on changes.

DevOps and the DBA: Conclusions

Perhaps the most frustrating part of becoming a DevOps DBA and implementing these core values in your IT organization is the startling lack of specificity: There is no specific set of approved tools for technically creating a DevOps environment, nor is there a specific process that works with every organization to break down the barriers between groups. Also, there is no specific set of metrics to deploy that will automatically and definitively measure if your organization has reached the pinnacle of DevOps implementation.

In addition, there’s no specific roadmap for a DevOps DBA to follow to implement the core values of DevOps; instead, it will depend on many intrinsic and extrinsic factors. It may require relatively small changes within your IT organization, like having stakeholders from the various teams within the organization sitting in on team meetings so that everyone contributes and supports each other. Likewise, automating software deployments will depend upon what tools are already in place within your organization and how much effort will be required to integrate those tools throughout the DevOps team. And deploying the resources necessary for development, unit testing, system and integration testing, and especially regression testing may be facilitated through a cloud-based environment like Oracle Public Cloud or an on-premises Cloud deployment (think: Oracle Cloud at Customer) that allows trusted developers and managers to orchestrate, provision and control a new environment without having to wait on system administrators and DBAs.

In conclusion, DevOps at its essence is an approach that promotes closer collaboration between lines of business, development and IT operations with the goal of quality rapid IT service delivery. Oracle DBAs who realize the advantages of this new delivery model and can leverage it as they transition to the role of DevOps DBA are therefore well-positioned for the future of our industry.