Instrumented SSH on NERSC Systems

NERSC uses a modified version of SSH on all of our systems that allows us to record and analyze the content of interactive SSH sessions. 

Why are We Doing This?

Credential theft represents the single greatest threat to security here at NERSC.  We are addressing this problem by analyzing user command activity and looking for behavior that is recognizably hostile.

Until SSH came into widespread use, it was trivial to monitor login sessions and analyze them for mischievous activity.  Furthermore, this kind of intrusion detection proved to be very effective with few "false positives".  Using this version of SSH at NERSC, we are simply recovering that capability. However, we recognize the importance of being candid about this to our user community given the assumptions normally made about using SSH.

The data collected with this version of SSH is sent to one of our security systems where it is analyzed by an intrusion detection system called Zeek.  Using various signatures, some complex and some fairly simple, Zeek is able to alert us when an account appears compromised.  Furthermore, once a compromise is confirmed, the logs from this version of SSH will help us determine the extent of the compromise and what, precisely, the intruder did.

In addition, we have added a set of patches to SSH developed at the Pittsburgh Supercomputing Center. These patches improve the performance of SSH/SCP/SFTP, particularly when moving large data sets over long-haul, high bandwidth networks.  For more information on these patches, see the PSC site.

What Does This Mean for You?

Any time you are logged in to a NERSC system via SSH, most of your keystrokes as well as anything displayed on your screen from our system will be recorded and analyzed by our intrusion detection system.  This recorded information may include any potentially sensitive information such as passwords.  Of particular importance to understand is that if you ssh from a NERSC system to another institution, that session will also be recorded.  For this reason, it is not recommended that you "step through" NERSC systems on your way to other systems.

It should also be noted that we are taking every precaution to protect the data we collect.  It is never transmitted in "clear text" and it is only stored on heavily protected security systems with extremely limited access.  While we attempt to filter passwords out of the data we collect, it's simply not feasible to do this with 100% reliability.