In 2020, many businesses were forced to take on remote work during the COVID-19 pandemic. Remote work was a new world for many, and to no surprise, brought on several challenges. Jack Wood and Dennis McCormick, two City & County of San Francisco, CA Controller’s Office employees, were joined by Doris Wong, CEO of CCSF’s implementation partner Smart ERP Solutions, to share what it takes to implement a PeopleTools upgrade and HCM/ELM catchup, all remotely during COVID-19.

About City & County of San Francisco (CCSF)

City & County of San Francisco is the only combined city and county in California. There are approximately 36,000 employees across 55 departments and 35 labor unions that serve CCSF. There are 15,000 suppliers and 500,000+ payments annually. For reference, CCSF’s budget was $11 billion in the 2018-2019 fiscal year.

CCSF’s enterprise applications consist of the following modules (which have been internally branded):

• PeopleSoft Interaction Hub (iHub) – Branded “SF Employee Portal”
• PeopleSoft Human Capital Management (HCM) – Branded “SF People & Pay”
• PeopleSoft Enterprise Learning Management (ELM) – Branded “SF Learning”
• PeopleSoft Financials & Supply Chain Management (FSCM) – Branded “SF Financials” and “SF Procurement”
• Cognos Budget & Performance Management System (BPMS) – Branded “SF Budget”
• Oracle Business Intelligence Enterprise Edition (OBIEE) – Branded “SF Reports & Analytics”

Additional applications include Cognos Business Intelligence, Cognos Informatica, Oracle Identity & Access Management (IAM/IDCS), TimeLink, Control-M, Phire Architect, DataGuard, and GoldenGate.

CCSF’s Vision & Strategy for PeopleSoft

The mission of CCSF’s Controller’s Office is to ensure the City & County’s financial integrity and promote an efficient, effective, and accountable government. The Controller’s Office has several strategic goals, including:

• Promote best practices and accountability in City government
• Support informed policy decisions
• Safeguard the City’s long-term financial health
• Provide high-quality financial services
• Support the City’s financial systems and infrastructure
• Sustain the City’s financial operations in a disaster
• Increase public access to useful and timely information
• Invest in and value its employees
• Manage the Controller’s Office effectively

Within the Controller’s Office, there are eight divisions, one of which is the Systems Division. This division focuses on the Controller’s Office’s IT systems, including PeopleSoft. The focus over the last few years has been staying current and efficient with Images, PeopleTools, and releases. This focus is so vital, that CCSF did an exercise that helps embed it into its core values and mission:

• Where We Want to Go: We envision being a premier IT organization and trusted partner within the City, as well as a national leader in enterprise systems implementation and support.
• Purpose: Continuous improvement within our people and with our business solutions; consistently pursuing best practices and innovative technologies; be efficient in staying current with application and security updates; provide work that has diversity for those looking for new challenges, consistency for those looking for balance, and opportunities for those looking to grow.
• Operating Principles: Value and recognize the good work of others; prepare for and run effective meetings; lean in and contribute to conversations; delivered or configurable solutions first and customize only when necessary; communicate to manage expectations of customers, peers, and supervisors; assume positive intent; have fun and make it a better place to work.
• Harmonizing Values: Collaboration; happy staff; reliability; operational excellence; outstanding customer service; ownership; openness.

CCSF’s Get Current Strategy

CCSF’s Controller has had a vision laid out over the last 10 years that included updating all of the city’s backend systems. CCSF started in 2012 when it went live on its HCM system and retired mainframe legacy systems. They also rolled out ELM, Portal, and iHub during that period and opened up to all of the 30,000 users. The major update was replacing the FAMIS mainframe system in 2017 that had been in place for 20+ years. It was replaced with PeopleSoft FSCM and OBIEE. CCSF is also in the progress of replacing a legacy Budget system.

As of 2018, CCSF had only undergone one major upgrade project. In 2019, the team took a look at what the vision needed to be moving forward and decided to emphasize the importance of staying current. The decision was made to do one update per year on each system, which started in 2019 on the Financials side. In November 2020, CCSF upgraded to FSCM Image 35 and was able to greatly improve its process. You can see the comparison in the chart below.

CCSF did the hard work to retire legacy systems, so now staying on the Continuous Release model will help them stay on the path toward avoiding outdated systems. CCSF credited much of their success to implementation partner, Smart ERP Solutions, who runs the core upgrade for CCSF. The CCSF team feels that outsourcing is the right model to help them to upgrades faster, cheaper, and easier for all.

COVID-Delayed HCM/ELM Release

CCSF’s plans for a March 2020 go-live for HCM and ELM were put on hold when COVID-19 hit, but the team was able to bounce back quickly and reschedule the go-live for late April 2020. Reasons for the delay included:

• San Francisco’s mandatory shelter-in-place was implemented on March 17, 2020.
• Staff had to adjust to working from home (dealing with VPN, moving equipment, etc.)
• CCSF is a Public Sector organization, so there was an influx in new work to help support the City’s COVID response. The staff was working full time at the Emergency Operations Center and on building new systems tools.
• Uncertainty!

Some of the factors the drove the selection of the new date were:

• Aggressive Approach
  • Minimize throw away work as CCSF introduces new code to Production
  • Needed to do another MTP pass – more time/money
  • If not April, it would delay other projects. Then, go-live couldn’t be until November at the earliest (more likely March 2021). Not ideal!
• Conservative Approach
  • Given everything CCSF needed to do to support employees and the community, why take the risk of something going wrong?

While there were undeniably risks of doing a major Image catchup (Image 13 to Image 32 catch up) during COVID with so much else happening, the CCSF team ultimately decided that it was worth the risk to get this project done sooner than later. The real challenge came from doing the work remotely while handling the HCM upgrade (Image 13 to Image 32), a big PeopleTools jump (8.54 to 8.57), a highly-customized HCM system (90 retrofits/new), an ELM upgrade (Image 13 to Image 19), new tiles/links/slide outs for iHub, and new Expense functionality for FSCM concurrently.

When planning the downtime scenarios, CCSF’s best case was estimated at 56 hours (noon on Friday to 8:00 pm on Sunday). The worst-case was 72 hours (noon on Friday to noon on Monday). The team was very close to the best-case scenario in real life, taking 59 hours to complete (noon on Friday to 11:00 pm on Sunday).

The remote project team was made up of three Smart ERP upgrade specialists, a project manager, a CTO, a project director, five PeopleSoft Administrators, four developers, 20 functional team members, four DBAs, three Server Administrators, two people working on network, and three people working on IAM/security. While the team faced minor local internet and DRP issues, overall, the go-live was a success.

What Was Done Differently Due to COVID

Due to the project being done remotely, CCSF had to approach it differently, which included:

• A fully remote upgrade
• More contingency planning than normal due to uncertainty
• Heavy reliance on collaborative tools like Smartsheet and Teams
• More QA in advance. Really tried to not use the delay to introduce scope focused on QA testing.
• More extensive and organized smoke testing scheduled

Results of the Remote Upgrade

For the end users:

• Escalation calls were scheduled for the entire week but were canceled on the second day
• All critical issues were resolved by Tuesday (implementation finished late Sunday before)

For the CCSF team:

• Harder to do due the number of communication channels and sidebar conversations in real-time
• Lost local internet access at the command center
• Encountered some last-minute challenges and had a communication lag that created stress

Despite challenges, the team was able to pull it off!

Lessons Learned

CCSF shared some of the lessons learned from this remote upgrade project:

• Remote upgrades are possible and may be the “new normal.” Start planning for them now!
• Know the tools you are going to use in advance. Don’t introduce new things that weekend!
• The core project team will be overwhelmed with update requests. Staff (or even overstaff) the communications function.
• Remote locations introduce more complexity and uncertainty. Try to mimic go-live during your dry run or tabletop exercises to test tools and access.
• Get alignment from your executives to ensure full buy-in on why you are doing it in advance. Be clear about the risks and benefits.
• Test more. QA prior to go-live and have more extensive smoke testing.

To learn more about CCSF’s remote upgrades and Continuous Release plan, check out the additional resources attached below.