As part of PeopleSoft Industry Day 2019, representatives from the University of Colorado spoke about how the university handles load testing. Load testing is critical to the success of The University of Colorado, and the University Information Services team makes heavy use of load testing technology for both production readiness and troubleshooting in both production and non-production environments.

During the presentation, the group walked through four types of testing, the importance of identifying success criteria, identifying what you’re trying to test and measure, integrating different types of testing, and more. Dive into the what, how, and why behind load testing at the University of Colorado!

4 Types of Testing

When it comes to testing your system, there are four primary types of tests to consider:

1. Baseline
2. Load
3. Stress
4. Failover

A baseline test is a preliminary test execution to collect metrics for later comparison. These are most useful when you know changes are coming such as PeopleTools Upgrades, PUMs, or changes to functionality. A series of results from baseline tests can help establish a norm as well as identify trends.

Load testing involves applying a pre-determined load to a system and measuring the system’s reaction. In addition to software systems, load testing is used in building bridges, airplanes, and thousands of other things.

Stress testing is adding continual, increasing, measurable load to a system until the system breaks.

Failover testing verifies a system’s ability to handle or absorb individual component failures and continue running. For instance, while the load test is running stable at full capacity, you can remove and replace parts one at a time. These parts may be IH web, IH app, ICS app, ICS web, or IGW.

2 Testing Approaches

As you create a plan for testing your system, you will choose between a monitored and scheduled overnight test. Monitored tests are typically run for two weeks. These can be baseline and custom tests and will be dependent on the QA person, PS admin, portal developer, and infrastructure. Scheduled overnight tests are focused and repeatable. They can be executed weekly, unmonitored, and run overnight after stage refresh. The test team will review the results the morning after. Testing environments and tools can contain various business divisions or campuses, many people, multiple systems, and different software.

Load Testing at the University of Colorado

University Information Services (UIS) is geographically and technologically dispersed across campuses in Colorado Springs, Boulder, Denver, and Anschutz Medical Campus. The university employs 35,000 people and teaches roughly 65,000 to 70,000 students. The university’s system layout includes standard DEV, TST, STG, and PRD with special projects called SP1, SP2, SP3, and SP4.

The UIS PeopleTools versions are as follows:

• IH: PeopleTools 8.56.14
• HCM: PeopleTools 8.56.14
• FIN: PeopleTools 8.56.14
• OSB: PeopleTools 8.56.14
• IGQ: PeopleTools 8.56.14
• ICS: PeopleTools 8.56.14

For load testing, UIS leaders chose to compare Silk Performer against 19 Admin Server. The test included 49 virtual agents with 18 to 20 virtual users each. Critical business moments were semester startup (ICS and IH), open enrollment (HCM and IH), and major upgrades (all PS).

To determine whether or not the test was passed, the team identified success criteria including error rates and page load times. For UIS, any error rate under 6 percent is acceptable during a load test. Page load times are highly variable and depend on page complexity, network, hardware, and other things. The targeted student page load time is under eight seconds.

Most UIS load tests occur in a Stage environment. This takes place for PeopleTools upgrades, critical patch updates, and major functional changes like open enrollment. However, the team does execute two production load tests per year—one before fall startup and one before spring startup.

This load test contains three scenarios:

• Scenario 1: AJAX Campus Portal 900 virtual users with OID (66% UCB / 15% UCCS / 19% UCD)
• Scenario 2: AJAX Campus Mobile 150 virtual users with OID (66% UCB / 15% UCCS / 19% UCD)
• Scenario 3: Failover test (IH Web, IH App, ICS App, IGW, OSB, ICS IBA)

Defining Load Testing

Follow these points to help you define your test:

• AJAX vs. “Headless” Scripts
• Virtual Agent Load (90% CPU) vs. Server Load
• Open Enrollment
• Scenario 4: HROW 400 users with OID by using IE
• Success Criteria:
  • Generate 600 Concurrent Users
  • 90 percent of pages load < 10 seconds
  • Plan elections match with the number of input file users

Executing Load Testing

Follow these points to help you execute your test:

• Pre-work
  • Configure Test Users (8000) and load into Directory
  • Did you clear your Web and App Server Caches?
  • Verify Scripts via Single User Test
  • Verify Virtual Agents via Single User Test (also populates cache)
  • Verify Test User Authentication (all 8,000 users)
• Ready? Everybody Ready? Go!
  • Skype Persistent Chat Window
  • QA notifies the group of key events
  • Run to completion, gather results, set up for the next test

Why Load Testing Matters

In addition to testing known potential problem areas, load testing reveals issues you never knew you had and things that functional testing would never find.

For instance, in the fall semester of 2016, UIS discovered the 57th-minute issue. During a standard load test for a PeopleTools ‘dot’ release update, they noted an error spike at the 57^th minute past the hour. At this point, users would experience a loss of access to the system. The IT team knew no root cause, and issues could be accidentally migrated to PROD and result in service interruptions. With only one week until the system would need to support 190,000 portal logins per day and peaks of over 20,000 logins per hour, the team urgently set out to find a solution.

After shutting down all non-essential services, the team adjusted the load balancer, portal content, campus solutions content, and virtual agent servers, but the issue persisted. Then, a log file analysis from the virtual agent server yielded a new suspect: The Adobe Flash updater. The updater would launch at 57:01 as a scheduled task on a system account. It didn’t show up when they looked at task scheduler or services like the QAUser account. The team needed to disable the two adobe services and disable the scheduled task on all virtual agents for the Sunday load test.

In the end, it took 20 people and 369 manhours to troubleshoot the issue over a timespan from August 3 to August 12. UIS is proud to say the hard work paid off, as fall startup and subsequent load tests went by without fault.

Final Pieces of Advice

As other companies initiate load testing, the UIS team recommends special consideration over whether you’re simulating the right load, choosing between client load and server load, choosing between read-only and dynamic testing, server recovery during failovers, and whether or not your test is real world. They also keep in mind that production is like an organism; things are always changing and there are constantly moving targets.