Is CCSF’s network safe? The final answer – A Bug in the System, Part 3
By Joe Fitzgerald
The Guardsman
(( NOTE: SCROLL TO THE BOTTOM OF THIS ARTICLE TO SEE A COLLECTION OF RESEARCH AND DOCUMENTS REFERENCED IN THIS ARTICLE ))
A Bug in the System is a multi-part article chronicling an alleged major virus attack on City College’s computer systems. “Part One” ran in the Jan. 25 issue of The Guardsman. You can read the article on TheGuardsman.com.
First reported at a City College board meeting in January, the allegations of viruses and security breaches at City College have spawned countless questions and accusations within City College’s Information Technology department and the school administration.
The ultimate question being this — is your personal data at City College safe?
The answer relies on whether or not you believe a report filed by external “digital forensics” auditor USDN, inc., is accurate.
The reality painted by USDN’s report describes the entire school’s network’s infrastructure as vulnerable to attack by a host of malware and viruses, such as the high-profile “Ghost-clicker,” and that there are definite steps the college needs to make to tighten its security.
Their report has come under fire from the college community as wildly inaccurate and based on methods that amount to guesswork and charlatanism. The college community alleges many things about the auditor, alleging that the $78,000 spent on the company’s consultation findings was money wasted.
GETTING PERSONAL
To understand the allegations against USDN, you have to go back a ways into the technology department’s history with their relatively new Chief Technology Officer, Dr. David Hotchkiss. As previously reported by The Guardsman, the college community has launched a witch hunt for the CTO.
Many of their misgivings with him began when he started reorganizing the department, causing grievances with many employees who attested to The Guardsman that they had their feathers ruffled. But as the Hotchkiss continued his work, it was clear that the friction between him and his staff of more than 70 was beginning to sully his reputation at the college.
The Guardsman spoke with more than ten employees who described working under Hotchkiss as an exercise in distrust and fear.
Recently one employee, who wishes to remain anonymous, revealed a detailed log of grievances, including that Hotchkiss issued disciplinary action against an employee who “embarrassed him,” and frequently took credit on his City College-hosted blog for work that others performed.
The log cites Hotchkiss as changing meeting minutes to protect himself against actions by the Service Employees International Union, which represents his staff. Most importantly, many say the CTO speaks to few, if any employees, and is only open to people who attain a certain level of status in the administrative hierarchy.
Basically, they say, he only speaks to you if you “matter.”
Not all people in the college hold the same opinion, however. In an interview in February, Computer Science department Chair Craig Persiko said that he felt most of the friction has revolved around Hotchkiss’ focus on security, versus the aims of the college to support uninhibited academic freedom.
Benton Chan, information systems engineer assistant, chalks it up to a difference of cultures.
“He’s from a military background,” Chan said. “He was unprepared for the college’s political institution.”
And Hotchkiss agrees.
“You have to understand my background as a military guy. The colonel reports to general… it’s defined,” Hotchkiss said. “The academic senate doesn’t have any control over my career. I will always go to my boss, and my boss’s boss.”
The reason this has any bearing on the virus situation at all is this: when the security breach of City College’s network was first reported, Hotchkiss had zero credibility with his staff or the college community.
No one believed him.
WITCH HUNT
When Hotchkiss first presented the reality of the security breach at the FIT committee in January, it did not go nearly the way he thought it would.
Hotchkiss said that Trustee John Rizzo called him up two days before the meeting to tell Hotchkiss that he put his report on the agenda. The CTO’s presentation turned up in an SF Chronicle news article that gave the college a media relations black eye.
Hotchkiss claims that Rizzo “threw him under the bus,” and set him up to present his arguments to the meeting in a way that would alert the Chronicle.
Rizzo denies the allegation. “That’s utter nonsense,” Rizzo said. They had settled on the agenda presentation weeks before, he said.
Who actually contacted the Chronicle may never be known. What is known, however, is what happened next.
The data that Hotchkiss used to present at the meeting was demanded up, down, left and right for weeks. Faculty and IT staff made public comments against Hotchkiss at board meetings. The SEIU formally presented the college board a petition with over 500 signatures against the CTO. IT staff filed public records requests, demanding the CTO hand over the data proving his assertions.
Hotchkiss didn’t provide the initial data to any IT staff, and amidst this cloud of distrust USDN went to work gathering further evidence of security breaches in City College servers.
In a packet of documents he sent to the Board of Trustees, Hotchkiss said, “my silence is standard industry practice for this type of incident. The taking of any personal information… is a serious event, a cybercrime. I do not discuss such things.”
Seeing USDN as an extension of Hotchkiss, the College Advisory Council and then-Chancellor Don Q. Griffin requested that Peter Goldstein disconnect USDN from the college’s server just as they were collecting the last of the evidence on the security breach.
This halted USDN’s ability to collect forensic data they say the FBI is waiting to see. Moreover, the device that was collecting the data was not properly disconnected, filling its “cache” or memory with loads of useless and redundant data, Tony Castillo, USDN’s CEO, said in an interview with The Guardsman.
According to the packet that Hotchkiss sent the board, this was a willful destruction of evidence of a crime. Hotchkiss’ claims don’t end there.
His packet of information also alleges that Vice Chancellor Peter Goldstein used “unqualified personnel to evaluate a scientific report, delayed and interfered with the cyber incident investigations,” and “ordered a violation of statements made to the Department of Education and Accreditation Team.”
He also accuses Network Administrator Tim Ryan of similar acts, as well as points a finger at Griffin, essentially saying that Griffin intentionally delayed USDN ability to create their report because he feared the results would go public.
Goldstein hadn’t read the packet before press time but said, “If those are the allegations, they are completely wrong, and very reckless.”
Ryan also denied the allegations, and added, “our chancellor is very ill and very respected, I don’t think [Hotchkiss] should be making allegations.”
The cover letter of the packet Hotchkiss sent to the Board of Trustees:
THE FINDINGS
Ultimately, USDN came to the conclusion that the college did not have enough technology in place to protect itself from the types of malware that are attacking the college. One of the school’s two firewalls, called “Palo Alto,” meant to guard the school against incoming malicious attacks, is currently being overstretched and overused, Castillo said. An email from the college’s firewall vendor, Dataway, confirmed this.
The college’s response was to immediately refute the findings of USDN’s report, the “Executive Report of Network Incident Findings,” which details 81 instances of personal information being sent out in digital transmissions from the college.
Castillo wasn’t surprised by the reaction. “Breaking news flash! The audit target doesn’t agree with the findings,” he said.
But when it comes to the basis of his report, he said that the basic assertions made of the IT department’s lax way of handling internet security is seconded in an independent report made by Dataway three years ago, as well as an internal audit of the IT department performed by college employee Ric Jazaie.
“They’ve had four audits. Not one, not two, but four,” Castillo said. The fourth audit, performed by Semantic, couldn’t be verified by press time. Jazaie agreed with the assessment that his audit of the IT department was in agreement with the Dataway report, but hadn’t seen USDN’s findings.
When long time IT employee Shirley Barger had a look at USDN’s report, she found much to argue with.
Barger said the report is based on incorrect assumptions, failing to take into account the fact that one of the servers sending out the alleged personal information is a spam filter, whose job is to spit transmissions back out where they came from. She also said that the IPs that the security system USDN provided blocks internet addresses indiscriminately, and may cut off international students’ access to City College’s website.
Castillo said that server, known as Name Service 7, was taken into account and not included in the USDN report of personal information leaked from the college. He also said that the software he provided, Alien Vault, actually looks at the content of outgoing traffic first, before blocking an internet address. In this way, you control for “false positives,” that many in the college have been saying the USDN report consists of.
The problems that exist with NS7 and other servers at the college, Castillo said, have international implications.
The initial report (minus appendices) that USDN made to the Board of Trustees:
FBI NOTIFIES CCSF
The FBI contacted City College in a letter dated January 27, 2012, concerning what Castillo said is one of the school’s many malicious attackers – ”Ghost-clicker.” The FBI notified the college that they were one of 4.2 million infected computers worldwide.
The virus is more officially known as a “DNS changer,” which essentially directs the infected computer to conduct ad-hijacking and click fraud. It redirects the user to alternate advertisements that are on a webpage to generate millions of dollars for the hackers.
The problem, Castillo said, isn’t just that City College students are clicking to generate money for these hackers. It’s that City College is almost like a Ghost-clicker hornet’s nest that sends out copies of the virus to other computers across the country.
An FBI supplied website shown to The Guardsman by an ITS employee showed how easy it is to verify if a computer has Ghost-clicker. The employee said thousands of computers are now clean.
This reaction-based method of solving security problems is part of the problem, Castillo said.
NOTHING TO SEE HERE
“The problem isnt the PII in of itself, its just a symptom. Everyone has PII breaches,” Castillo said. PII stands for Personally Identifiable Information, like social security numbers or driver’s licenses.
“The problem is the way they’re dealing with it, the policies and the responses to the incidents. They refuse to accept these incidents are occurring. If you never recognize the system you’ll never ever, discover the root cause,” he said.
Denying the problem seems to be something the college is intentionally planning to do. A draft letter titled “The ITS Department is on Top of Security” has recently surfaced. The letter acknowledges a security breach in February 2007. However, the letter goes on to say, “There was a concern that some of our servers… had been compromised….Fortunately, it all turned out to be a false alarm.”
The letter then cites instructor Sam Bowne, who says “The CCSF-network is well-maintained and much cleaner than comparable networks at other organizations.”
Hotchkiss said that Sam Bowne has been under investigation for cyber bullying, as well as inciting a leak of a City College instructor’s personal information.
In a Tweet, Bowne wrote, “I ran all those tests, nothing malicious found. If it’s evil, it’s smarter than me, but i’m a n00b at malware analysis.” “N00b” is internet slang for newbie, meaning an amateur.
The draft letter of CCSF denying a problem of recent security breaches:
THE BUG IN THE SYSTEM
The witch hunt for Hotchkiss, no matter how warranted, may be distracting the IT department and the various technology committees, such as the Information Technology Policy Committee, from carrying out any further meaningful changes in network security policy in the school.
Amidst all the finger pointing, bad blood, shouting matches and allegations of federal offenses, the one person who is respected by every party involved is Information Systems Engineer Benton Chan. Chan wears many hats and is one of the tech folks most involved with security at the college.
In Chan’s opinion, the real culprit, responsible for more breaches than any amount of external attacks, are the computer users of City College.
Focusing on technological processes is fruitless, Chan said. It would be better to simply educate users on how to behave better while using technology.
Chan raised his finger up, pointed it straight ahead and said with certainty that the real crux of the matter is “policy, not process.”
In other words, the “bug in the system” is us.
RELATED DOCUMENTS AND RESEARCH:
Below is a collection of the highlights of the research done for this piece. Contained are most of the documents from Dr. Hotchkiss’ packet to the City College of San Francisco Board of Trustees, where he outlines accusations against Vice Chancellor Peter Goldstein and Technical Operations Manager Tim Ryan. Also included is a draft document by the college asserting that there was “a false alarm,” instead of a threat of security breaches. Click on any document in the shelf to browse through it, and you can scroll through the shelf as well.