During the DEFCON16 hacker conference in early August this year there was a presentation on Hacking OpenVMS. Presentations about OpenVMS have not been seen at hacker conferences for many years. In the presentation security researchers from signedness.org presented two security vulnerabilities of the OpenVMS Platform.
Around the same time as DEFCON16 a report was released on the bugtraq distribution list (http://seclists.org/bugtraq/2008/Aug/0056.html) describing a vulnerability which allows remote escalation of privileges with the Finger daemon with Multinet. Process Software Inc has released a patch for Multinet which fixes this problem. However, best practice for publically accessible systems is to disable the Finger client and server.
The presentation at DEFCON16 described two vulnerabilities and generally discussed hacking OpenVMS.
One is a local exploit using a remote (and malicious) Plan File to escalate privileges within the Finger client. Multiple versions of the HP TCP/IP Services for OpenVMS Finger client are vulnerable. No patch is available for this issue. Other vulnerabilities are known in the Finger software. The recommendation is to disable the Finger client and server, as any sensible system manager would do.
The second is a stack corruption vulnerability within a widely used system run-time library routine which can be exploited when that routine is being used in a program installed with privileges. The exploit would typically be used to execute code supplied by the hacker as part of the attack. A successful attack using this mechanism could allow a local unprivileged user to gain privileged access to the system. HP have released a patch for currently supported versions of OpenVMS which fixes this vulnerability (SMGRTL-V0100), however the issue does exist in other versions of OpenVMS. Versions for which the fix has been released are: OpenVMS Alpha V8.3, V8.2, V7.3-2, and OpenVMS Itanium V8.3, V8.3-1H1, V8.2-1
The security researchers have said they had no previous knowledge of OpenVMS and used the usual techniques that they use on other platforms to search for vulnerabilities (eg: techniques such as "stack smashing"). They informed HP of the details of the vulnerabilities weeks before the presentation and no detailed exploits have been released publicly.
The DEFCON16 presentation is now available on the internet and the vulnerabilities have been extensively discussed on the OpenVMS newsgroup (comp.os.vms). The class of vulnerability found has been known on other platforms for a long time. OpenVMS has not been targeted by security researchers for many years but now more probing of OpenVMS security should now be expected. OpenVMS is significantly different to more widely known operating systems and was designed from the beginning to be robust and secure. However flaws in native software or in software ported from other platforms can still open security holes. Proper management of OpenVMS systems will always reduce the risk. OpenVMS remains one of the least vulnerable operating systems in existence.