The IT world has been in shock as we’ve all learned about the Solarwinds hack.  Network and server monitoring software inhabits a special niche where it often has full access to the servers and devices on the network in order to monitor all internal resources.  Because of this it needs to be as robust, safe and trustworthy as possible.   Given the fall out of the Solarwinds hack, customers have been asking us about our product safety.  We’d like to assure you our software is safe and explain the steps we take to keep it that way.

When we talk about our products, it’s useful to split the files that make these products into two types of interest – those that are executable (.EXE, .DLL and .SYS files) that we produce, and executables from other vendors.

Executables from Other Vendors

We have recently completed an audit where we have checked every single executable file that we get from vendors to ensure they haven’t been tampered with.   Some of them were already Signed by the vendor so we know they are OK.   Others we have tracked down back to their source and compared to ensure they haven’t been altered.  We keep all binary files in our source control solution (something not all companies do) so we can audit and see if any of them ever change.   For the 3rd party executable files that were not already signed, we have done a one-time signing of them now that we have audited them and found them matching the originals.

Our Executables

Our executable files are those that we compile from our source code.   There are a very few that are not compiled at the time of each build – those were signed the last time they were built, and then not touched again.   For the files that are compiled each time, the .EXE and .SYS files have been signed for a number of years.   Starting now, we are also signing all DLLs that we compile.

According to an analysis from Microsoft, Solarwinds was actually hacked at the source file level.  Hacks such as this require obfuscating certain bits of the code to hide critical details, such as external hostnames that will be contacted.  Sources files written in .NET languages have a large library of possibilities for obfuscation such as compression and encoding/decoding data.  We don’t use very much .NET in our products, so were able to look at every one of our .NET source files manually to check for correctness, and nothing unexpected was found.    Note that in the Solarwinds case 4000 lines of new source code had to be introduced for the hack, so it would be easy to spot manually.   For C/C++ code, extensive built-in libraries don’t exist and have to be added manually.   We’ve ensured no additional libraries have been added, and then checked all code that does compression or encoding/decoding of data and found nothing unusual.

Build Checks

Now that all 3rd party executables are signed (either originally by the 3rd party or one-time by us), and all of our executables will be signed at build time, our build system will verify all executables are properly signed as part of the normal build process.

Our Build System

Our build server is only accessed by one person.   That server has our AD Login Monitor on it so we can see who accesses it.   In addition, all code is in our source control system.   One nice side effect of being a smaller company is we all know each other and know what each other are working on.  Any unusual file changes would stand out and attract attention because each time source files are committed to source control, a notification goes out to the Slack channel letting all developers stay aware of changes.

In addition, we have our File & Directory Change monitor watching files that shouldn’t be changing so we’ll receive alerts if they do change.

Network Security

To help prevent attackers from getting into our network in the first place we have a number of protections.  Anti-virus applications are used and pattern files are kept up to date.   Windows Updates are regularly installed on all computers.  Windows Firewall is on and running on all computers.  We use two-factor authentication for remote access, and each user can only access their own workstation.  Of course, being a monitoring company means our software is on multiple servers (test and production) which provides overlapping monitoring and alerting of all systems.   We are taking this opportunity to further enhance our internal monitoring and have setup additional alerts watching for suspicious changes.

The post Solarwinds hack - Our Safety Measures first appeared on Network Wrangler - Tech Blog.