One incredibly powerful feature of the Go language that hasn’t seen a lot of use is its native support for virtual file-systems. Most C-style code is written using the standard file operations that all work on passing an integer “file descriptor” around. To re-purpose that code to use anything other than the OS-provided file would involve changing every file operation call in all of the source and libraries involved.
A “file” in Go is just another set of interfaces that you can replace.
The cool thing about that is that all Go code ever written to do operations on files will work just as well if you pass it some other thing you made that implements the file-system interfaces. All you have to do is replace the standard file-system implementations you’re used to, mostly provided by the os package, with your new virtual file-system, and all the ReadFile, OpenFile, etc. function calls will stay the same.
By using a couple of existing Go libraries that do the heavy lifting, it’s possible to implement a simple read-write virtual file-system that will be encrypted and compressed on disk in only a couple pages of code, as you’re about to see.
The two libraries we’ll use are:
rainycape’s VFS library – The VFS provided by the standard library is read-only. This library adds the ability to write to the virtual file system, and also already supports importing the file-system from several different compressed archive formats. We will simply extend this to add encryption, but as of this writing, rainycape’s VFS library is the only working implementation of a Go VFS with write implemented, so it will be the likely starting point for any other Go VFS project you might attempt.
bencrypt – An encryption utility package we will use just for the AES encrypt/decrypt functions.
|
1 2 3 4 |
import ( github.com/rainycape/vfs github.com/awgh/bencrypt ) |
The first of our two pages of code is a function that simply AES decrypts a compressed file with a given key and passes the decrypted compressed file into the rainycape VFS library, which provides a full read-write implementation of a virtual filesystem decompressed into memory from a compressed file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
// OpenAndDecrypt returns an in-memory VFS initialized with the contents // of the given filename, which will be decrypted with the given AES key, // and which must have one of the following fileTypes: // // - .zip // - .tar // - .tar.gz func OpenAndDecrypt(filename string, fileType string, aesKey []byte) (vfs.VFS, error) { f, err := os.Open(filename) if err != nil { return nil, err } defer f.Close() b, err := ioutil.ReadAll(f) if err != nil { return nil, err } clear, err := bencrypt.AesDecrypt(b, aesKey) if err != nil { return nil, err } bb := bytes.NewBuffer(clear) switch fileType { case ".zip": return vfs.Zip(bb, int64(bb.Len())) case ".tar": return vfs.Tar(bb) case ".tar.gz": return vfs.TarGzip(bb) } return nil, fmt.Errorf("can't open a VFS from a %s file", fileType) } |
The second page of code is to write your changes back from memory to the disk. This does the inverse of the OpenAndDecrypt operation, it just zips the file-system back up to an archive and then AES encrypts it with the given key.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
// SaveAndEncrypt converts the given VFS to the given archive type, // and then encrypts the archive with the given AES key. // Supported fileTypes: // // - .zip // - .tar // - .tar.gz func SaveAndEncrypt(fs vfs.VFS, outfile string, fileType string, aesKey []byte) error { bb := bytes.NewBuffer(nil) switch fileType { case ".zip": if err := vfs.WriteZip(bb, fs); err != nil { return err } case ".tar": if err := vfs.WriteTar(bb, fs); err != nil { return err } case ".tar.gz": if err := vfs.WriteTarGzip(bb, fs); err != nil { return err } default: return fmt.Errorf("can't write a VFS to a %s file", fileType) } cipher, err := bencrypt.AesEncrypt(bb.Bytes(), aesKey) if err != nil { return err } if err := ioutil.WriteFile(outfile, cipher, 0600); err != nil { return err } return nil } |
The final piece of code we’ll need to get around is a simple function that will AES encrypt any file. We can then make our starting file-system as a compressed archive and use this function to AES encrypt it to be loaded by OpenAndDecrypt.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
// EncryptFile will encrypt clearfile with aesKey and save it to outfile func EncryptFile(clearfile string, outfile string, aesKey []byte) error { f, err := os.Open(clearfile) if err != nil { return err } defer f.Close() clear, err := ioutil.ReadAll(f) if err != nil { return err } cipher, err := bencrypt.AesEncrypt(clear, aesKey) if err != nil { return err } if err := ioutil.WriteFile(outfile, cipher, 0600); err != nil { return err } return nil } |
That VFS interface returned by OpenAndDecrypt can be used just like the regular os and ioutil interfaces you’re used to getting from the standard library… and it will be easy to retro-fit your existing code to use an interface that could be a VFS.
Here’s a quick example of how you could encrypt a Zip file and then open it up as if it were the os or ioutil packages (because they both implement the same interface).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
aesKey, err := bencrypt.GenerateRandomBytes(32) if err != nil { log.Fatal(err) } if err := bencrypt.EncryptFile("fs.zip", "fs_test.aes", aesKey); err != nil { log.Fatal(err) } fs, err := bencrypt.OpenAndDecrypt("fs_test.aes", ".zip", aesKey) if err != nil { log.Fatal(err) } data1, err := vfs.ReadFile(fs, "a/b/c/d") if err != nil { log.Fatal(err) } |
Now, before you go typing all this into your favorite IDE, I should point out that all of these functions have already been added to the bencrypt library, so you can just call them directly from there.
IMPORTANT WARNING
Also, be aware that this simple method will not encrypt your files once they have been loaded into system memory. They could be read out unencrypted by a debugger, saved in a swap file, or a tool like Volatility.
Final Thoughts
To retro-fit your existing code that uses the os package, you’ll need a defined interface that you can use that could be bound either to os or to vfs. I would suggest stealing this one from the go-vfs package as a starting point.
The other kind of odd thing about this, is that it’s almost entirely a side-effect of how Go handles interfaces. Membership in interfaces is decided at runtime, based on whether or not the required functions exist in a type to satisfy the interface. This lets you define an abstract interface for a file-system after the fact that matches what os already does, and create a virtual file-system that implements that interface as well. Your existing code just needs to be repointed to your new interface instead of to os and everything else will work, even though your interface was written way later than the os package. The same trick could allow replacing or hooking other lateral aspects of standard library functions, that might yield similarly interesting results.