City College virus attack challenged – A Bug in the System (part 2)
By Joe Fitzgerald
A Bug in the System is a multi-part article chronicling an alleged major virus attack on City College’s computer systems. “Part One” ran in the Jan. 25 issue of The Guardsman. You can read the article by clicking here: Bug in the System (part 1).
Since Chief Technology Officer Dr. David Hotchkiss first revealed the virus attack against City College publicly in January, his findings and methods have been under attack by school technology officials of every stripe.
The story begins just shortly after the virus was discovered, in December of 2011. The new Chinatown/North Beach campus was nearing completion, and Dr. Hotchkiss thought the perfect fit for a 21st century green campus was a green computer system: ZeroClient.
ZeroClient is a system that uses ultra-thin, and ultra-expensive workstations that run from a single central server. Claiming that it could save the school money in energy and replacement costs in the long run, Dr. Hotchkiss launched a pilot program, putting a few of the computers in English and Business classes.
As positive feedback rolled in, Dr. Hotchkiss put in an order for ZeroClient machines for the Chinatown/North Beach campus at a cost of $750,000.
CLICK HERE FOR AUDIO: Dr. Hotchkiss presents the benefits of using ZeroClient hardware to the College Advisory Council on Dec. 1. Full recording available at http://www.ccsf.edu/Offices/Shared_Governance/cac.html.
In an email interview with The Guardsman, he said that he went directly to Joanne Low, at the time dean of the Chinatown/North Beach campus, to confirm the order. Dr. Hotchkiss relayed his interview by email after repeated attempts to contact him by phone and in person.
Only later did he discover that in doing so he failed to follow proper college protocol, putting in the $750,000 order without having it vetted by the college’s shared governance system. Shared governance allows representatives fromthe staff, faculty, student government and administration to weigh in on all important college decisions.
Dr. Hotchkiss said in an email interview with The Guardsman the system played no part in other requisitions he made for the Chinatown/North Beach campus, and just exactly what decisions shared governance makes and what it doesn’t lacks consistency.
“Process-wise I presumed that Dean Low had the authority to agree what technology would be suitable for the campus,” he wrote in the email.
In an interview, Low said she had no recollection of giving Dr. Hotchkiss express permission to order ZeroClient, and denied having the authority to allow him to do so in the first place.
According to accounts of eyewitnesses, and from available public recordings, when Chancellor Don Griffin learned that Dr. Hotchkiss had bypassed shared governance, he was furious.
“It’s a rogue operation, that’s what my problem is. It’s an assault on the shared governance process,” he said,scolding Dr. Hotchkiss at a December College Advisory Council meeting attended by staff and faculty.
“We’re dying here with no classes … if this falls apart, where do we get the money to replace it?” Dr. Griffin asked. “We have to then go into the general fund. And how do you get it done? We have to cancel classes and other services that are critical for students.”
The Technology Advisory Group, or TAG, was created for the sole purpose of helping Dr. Hotchkiss navigate City College’s rules and regulations as a new Chief Technology Officer. He is also required to attend regular technology meetings with the Information Technology Policy Committee, which sets tech policy at the school; his role as CTO is key at those meetings.
During a January ITPC meeting when he learned he was being recorded, which is legal in public meetings under state law, Hotchkiss abruptly walked out of the room and never came back. A recording of the meeting is available on the shared governance website at http://www.ccsf.edu/Offices/Shared_Governance/council.html .
The minutes from various meetings as well as interviews The Guardsman obtained show that Dr. Hotchkiss hasn’t attended any of the six TAG or ITPC meetings held since that day. He has also missed two recent CAC meetings, prompting the chancellor to have Hotchkiss called to explain his absence, according to members of the group. These meetings are also recorded and can be heard online through CCSF.edu on the shared governance page.
CLICK HERE FOR AUDIO: Dr. Hotchkiss discovers an Information and Technology Policy Meeting is being recorded, makes a statement, and walks out. Public meetings are allowed to be recorded by anyone under California’s “Brown Act,” a sunshine law. The full meeting can be heard at http://www.ccsf.edu/Offices/Shared_Governance/itpc.html. Academic Senate President Karen Sagnior and CCSF technology expert Tim Ryan can also be heard in the recording.
In the end, the ZeroClient system never proceeded past pilot stage, and the school never spent the $750,000. The incident shows however that Dr. Hotchkiss struggles with working within City College’s chain of command.
When the virus attack was revealed to the public in January 2012, pressure from college faculty and staff mounted against Dr. Hotchkiss.
Board of Trustees President John Rizzo and Hotchkiss knew of the problem since November 2011, but only disclosed it later. Rizzo said in an interview with The Guardsman that the time was used to gather more data about the true nature of the attack.
The specifics of the data would only be seen once by the general public, at a committee meeting in the form of a powerpoint presentation.
Hotchkiss broke the news of the virus at a January 12 technology meeting at City College. Addressing the chancellor and committee board, Hotchkiss said that college computers had been infected with computer viruses, putting students’ personal data at risk.
He also announced that the college had hired a local network security firm, USDN Inc., to investigate. Later on Vice Chancellor Peter Goldstein said that the FBI was also notified of the breach.
The next day, it was front page news in the San Francisco Chronicle.
The story spread nationally, appearing in the Huffington Post, ABC and NBC news, as well as tech publications such as Slashdot. The nation was discussing City College’s technological incompetence. Many college officials worried that this would hurt the school’s chances to raise funds in the near future.
Video coverage of the alleged virus outbreak by ABC 7 News. Similar reports aired on NBC, CBS and KTVU.
Even worse, college tech officials weren’t sure that the crisis was ever really that—a crisis.
City College technology experts interviewed by The Guardsman said they can’t be absolutely certain until they receive the final report, but early reports from USDN Inc. indicate that the virus was confined to a single computer lab used by international students.
Also there has been no conclusive evidence that any students have been victims of identity theft as a result of using City College facilities, Chancellor Don Q. Griffin said.
Not a single student has come forward to complain of identity theft.
When The Guardsman asked Hotchkiss if he regretted alerting The Chronicle to the attack, he replied, “I cannot regret an action I did not take.”
However, he added, “The college has a regulatory obligation to notify the potential victims of that crime. Wouldn’t you like to be notified if there was a potential your personal information was taken?”
Hotchkiss’ initial report and interview with The Chronicle suggested that tens of thousands of students were possibly impacted by the data breach. The college did send a general email warning students to be careful using college computers. To date neither Hotchkiss nor the school have issued any specific emails suggesting personal data was actually stolen.
Technology employees have stated in public meetings, such as the TAG/ITPC joint meeting in January, that they have asked to view his findings, but Hotchkiss has refused to let anyone see the data, even declining to distribute the PowerPoint presentation he used at the January meeting.
The Guardsman has filed Freedom of Information Act requests to obtain from Hotchkiss documents related to USDN Inc., as well as the information from his initial reports on the virus. Neither had been released at the time of this report, a violation of the 30-day limit of time the federal government gives for FOIA requests.
The Guardsman, having made California Public Records Act requests in person at the the college’s Gough street offices regarding contracts with USDN Inc., was told that Hotchkiss was refusing to release the data to the college’s legal counsel. City College officials, however, have been very open and giving with any documents that did not need to be obtained through Hotchkiss.
“I will not comment on any ongoing FBI investigation,” Hotchkiss has repeatedly said.
The FBI has confirmed that their Cyber Crime Division has been in contact with the school, but denies issuing a gag order to anyone at the college. The amount of information Hotchkiss wishes to reveal is up to him and the school, media relations officer Peter Lee said.
Hotchkiss’ direct superior, Vice Chancellor Peter Goldstein, denies asking him to keep quiet.
Hotchkiss’ silence is his own. And what his silence says about the severity of the virus outbreak and its legitimacy depends entirely on who you speak to at City College.
The lines are drawn
Throughout the crisis, one of the Chief Technology Officer’s’ most valuable resources has gone untapped in solving the security breach— his own department and its 70 employees.
In public testimony at various board meetings, numerous staff and faculty have criticized Hotchkiss for spending college money to hire an outside contractor, UDSN Inc., rather than turning to the vast team of technology specialists at his disposal.
At the Feb. 23 Board of Trustees meeting, Computer Science department chair Craig Persiko crystallized all the complaints that have dogged Hotchkiss over the past few months.
“I and many people on our committee feel that the allegation of this virus attack is overblown,” Persiko said to the packed meeting room. “We are especially concerned about the lack of communication from the CTO with his own staff. All the technology staff [first] learned of this virus infestation in the SF Chronicle.”
CIty College employees voice concerns about Chief Technology Officer Dr. David Hotckiss at the college’s monthly Board of Trustee’s meeting, January 26, shortly after the Chronicle article was released.
Persiko said in a telephone interview that he was especially concerned that Hotchkiss’ conduct was indicative of how he treats the Information Technology Services department. When asked what this says about Hotchkiss’ opinion of his own employees, Perisko said simply: “It implies he didn’t trust them.”
Shirley Barger, who has worked in the ITS department for over 20 years, verified that it has become increasingly difficult to work there, describing a hierarchical work environment where employees are not consulted on major projects.
Since Hotchkiss began heading the ITS department in 2010 seven employees have chosen to retire, including Glen Van Lehn, who was a network engineer from 1991 to 2010. For those of you keeping track, that means he has been working on the school’s technology since the era of dial-up modems.
He described to The Guardsman an atmosphere of disrespect in the department under Hotchkiss. When asked why he retired when he did, Van Lehn admitted that he retired early rather than work under him.
“He doesn’t listen to people. He simply imposes,” he said.
Many other college employees critical of Hotchkiss were unwilling to speak with The Guardsman, afraid that their public testimony could compromise the integrity of his annual performance review. The review determines the number of years, if any, to extend his contract.
Rebellion against the doctor
When The Guardsman asked Hotchkiss why so many in his department would respond negatively to him, he disagreed with the premise of the question.
“With new leadership comes new ideas and changes to the status quo,” he said in an email. “Many people don’t like change. However, most staff members have seen that the changes made are for the better.”
Athena Steff, president of the SEIU 1021 union that represents City College staff and all of Hotchkiss’ ITS department employees, presented a petition of “no confidence” circulated against Hotchkiss to the college’s Board of Trustees at their Feb. 23 meeting. The petition basically states that the SEIU believes Hotchkiss is not fit for his job as the head of the IT department.
The petition, signed by 500 people in the City College community, was delivered amid the roaring cheers of the 50 plus SEIU members in attendance.
When asked about the petition, Chancellor Griffin said he had never in his entire career at the college seen a petition of “no confidence” outside of normal evaluation procedures.
“What [Hotchkiss] has done is report a virus from one lab and issue a warning from his position as CTO. That’s his job,” Chancellor Griffin said. “What we’re determining now is if it’s egregious.”
At press time, USDN Inc. has finally given the college the missing second half of their report on the attack. It arrived as a torrent of data, lacking any summary or analysis, Vice Chancellor Peter Goldstein said.
That data has yet to be shared with college IT experts, but Goldstein expects that to happen within the next few weeks.
Tune in next month for the last chapter of our story: Part III, when the final reports of USDN will have been evaluated by City College tech experts, and the full amount of dollars spent on them will be made public, revealing the true nature of the “Bug in the System.”