OS X: cd /Volumes Returns Unknown Error 118

I spent most of the day helping a user fixed a problem with Xsan. The problem started last night. The user tried to create a share point, but the share point was not created successfully because the mount point was wrong. The share point was deleted afterward. However, the Xsan stopped working, and none of the volumes could be mounted.

The logs revealed that the Xsan could not create a directory in /Volumes for the Xsan volumes to mount. When I tried to change directory to /Volumes, I got this error:

-bash: cd: /Volumes: Unknown error: 118

And the mount command showed that there was a bogus entry for /Volumes.

map -static on /Volumes (autofs, automounted)

This entry was created by automount. I checked the configuration of automount(autofs) and did not find anything related to /Volumes . I suspected that automount got its info from Open Directory, but I checked Server Admin and Workgroup Manager and could not find the bogus mount point. There was no built-in utility for me to browse Open Directory.

I then found a java LDAP explorer: jxplorer. Using jxplorer was easy. I just downloaded it and ran the jxplorer.sh from the Terminal. I then connected it to the Open Directory server on port 389. It allowed me to explore my Open Directory structure. I finally found the bogus entry under /Mounts . The bogus entry was created when the user created the share point. Even though the share point was deleted, the mount entry remained.

jxplorer only helped me find the entry, but I could not delete it. I think you could use jxplorer to modify Open Directory if you provided a valid credential, but I did not figure out a way to enter that information. I had to use the command line utility dscl to remove the mount entry.

dscl command is not very intuitive, and there is a strange requirement when you try to delete an entry. You have to use double back slashes when you specify the path for the key. For example, in order to read a key, you can just use this command:

dscl /LDAPv3/127.0.0.1 -read /Mounts/myserver.com:\/Volumes\/server_volume

Note that I ran this command at the Open Directory server, so the LDAP server was at 127.0.0.1. If you want to delete a key, you have to use this command:

dscl -u directory_admin /LDAPv3/127.0.0.1 -delete /Mounts/myserver.com:\\/Volumes\\/server_volume

After comparing this command with the previous one, you will come to notice a difference between the two. You need to specify a user with directory admin privilege, and you have to use double back slashes to specify the key after  semicolon(\\/Volumes\\/server_volume).

After this mount entry was deleted from Open Directory, all the Xsan volumes could be mounted under /Volumes again.