In Community Server 2007 and earlier versions, the many file-related features (post attachments, site files, blog files, avatars, role icons, etc) all implemented their own file management logic and supported (a) storing files in the database (b) storing files in the file system or (c) configurable options allowing file storage in the database and/or file system. Adding new file-based features (in the core application or in add-ons) required implementing custom file management logic. Database-based storage was the default for post attachments. Write permissions needed to be granted in many locations within the CS application. Storing files in external systems (such as Amazon S3) required modifications to the custom file management logic used for each file-based feature. In general, there were some file management issues.
In Community Server 2008, the Centralized File System (or Centralized File Storage or CFS) was implemented to centralize file management (so there was one instance of file management logic used for all file-based features), support file storage providers (so that new file storage services, such as Amazon S3, could be used to store files), and to make it generally easier to add file-based features to Community Server (both in the core product as well as in add-ons).
The CFS sits deep in the Community Server API, between the API for file-based features and the configured storage providers:
Interacting with individual file-based features has not changed (or changed very little) between CS2007 and CS2008. The individual file-based features, however, use the CFS instead of their own custom file management logic.
In general, users and developers will not directly interact with the CFS. Only when new file-based functionality must be implemented is direct access to the CFS required. For example, to add/update/delete post attachments and their content, code should not access the CFS, but should instead access the add/update/delete methods of the PostAttachments class (which themselves interact with the CFS file store for post attachments). When implementing a new feature, such as video transcoding, it may be useful to have a new file store and interact with the CFS directly as a new file-based feature.
CFS Terminology
The CFS uses file store keys, paths, and filenames to store files:
File Store Key
The file store key is the name identified on the <fileStore /> node representing the file store in the communityserver.config file (defined later). It represents a unique set of files that are managed independently of other file stores.
File store keys must be between 3 and 255 characters and cannot contain any characters that are not valid in windows file system paths.
Path
Paths are similar to directories, but are technically used more like keys. Paths can represent multiple values separated by periods (the period is used for defining sub-paths and is also programmatically accessible via the CentralizedFileSystemProvider.DirectorySeparator property). For example, a common file system path may be "my\files\are\here" and the similar CFS path would be "my.files.are.here"
There is a technical difference between directories and CFS paths, however. When querying for files in a directory, files are generally accessed within a single folder, "my\files\are" for example. In the CFS, because paths are treated as keys, files can also be accessed using partial paths, such as "my.files.a", which would match files in the "my.files.are" path as well as the "my.files.and" and "my.files.any" paths.
Paths must be between 0 and 769 characters and cannot contain any characters that are not valid in windows file system paths.
Filename
Filenames are the names of the files stored in the CFS. They must be between 1 and 255 characters in length and must not contain any characters that are invalid in window file system file names.
File Stores in the CFS
Each file-based feature can use one or more "file stores" in the CFS. Each file store is defined within the communityserver.config file by a <fileStore /> node within the <CentralizedFileStorage /> node.
Each file store must define two attributes: name and type. Additionally, the downloadValidatorType attribute can optionally be set.
name
The name attribute defines the name of the file store (also referred to as the file store key) -- this is how the code used to implement file-based features will identify the file store when it interacts with the CFS.
type
The type attribute identifies the CFS file storage provider to use to store files for the file store.
CFS file storage providers are .net classes that inherit and implement the CommunityServer.Components.CentralizedFileStorageProvider base class (in the CommunityServer.Components.dll assembly). Community Server 2008 ships with two CFS file storage providers: AmazonS3FileStorageProvider and FileSystemFileStorageProvider.
Each file store can define its own file storage provider, with its own configuration. For example, post attachments could be stored in Amazon S3 but user avatars could be stored in the local file system. By default, CS2008 uses the file system file storage provider for all file stores and stores files in the "filestorage/" folder.
downloadValidatorType
Each file store can optionally identify a download validator. A download validator is a .net class that implements the CommunityServer.Components.ICentralizedFileAccessValidator interface (in the CommunityServer.Components.dll assembly).
A download validator is used to implement logic that identifies, for the file store, whether a file should be allowed to be accessed or not. In CS2008, only post attachments identify a download validator -- it is used to ensure that users have view and/or read permission to the attachment's associated post before allowing the attachment to be downloaded.
Additional Attributes
Each provider requires additional attributes to be defined on the <fileStore /> node. These additional attributes are documented in the comments just above the <CentralizedFileStorage /> node in the communityserver.config file included with Community Server 2008.
Creating New CFS File Stores
Creating a new CFS file store is as simple as adding a new <fileStore /> node to the <CentralizedFileStorage /> node of the communityserver.config file with a new name. When the <fileStore /> node exists, that file store can be used to add/delete/query files.
The existing file stores are named according to the full name of the class in the Community Server API that interacts with the file store. For example, post attachments are created/updated/deleted via the CommunityServer.Components.PostAttachments class in the CS API so the CFS file store used to store post attachment files is named "CommunityServer.Components.PostAttachments." I would suggest continuing this naming convention when adding additional file stores to the CFS.
Coding Against a CFS File Store
When implementing new file-based features for Community Server, I strongly suggest using the CFS for file storage. The CFS provides a simple API for adding, deleting and querying files and paths.
The following methods are exposed by the CFS (and must be supported by all CFS providers): GetFile, GetFiles, GetPaths, AddPath, AddUpdateFile, Delete.
To interact with these methods, code must first retrieve the file storage provider for a specific file store. This is done using the file store key, as in:
CentralizedFileSystemProvider myFileStore = CentralizedFileSystemProvider.Instance("FILE_STORE_KEY");
Which would load the file storage provider for the "FILE_STORE_KEY" file store (note that there must be a corresponding <fileStore /> node in the communityserver.config file named "FILE_STORE_KEY", otherwise, the Instance method will return null). The file management methods could then be called on the myFileStore object. Any operations preformed using the myFileStore object would be performed within the context of the file store named "FILE_STORE_KEY".
ICentralizedFile GetFile(string path, string fileName)
Gets the file store's provider's implementation of the ICentralizedFile interface representing the file with the given path and file name. If the file is not found, the GetFile method will return null.
ICentralizedFile file is an abstract representation of a file and supports retrieving the ContentLength, FileName, Path, and FileStoreKey of the file and also supports retrieving a stream of the content of the file (via the OpenReadStream method) and getting a URL to download a file (via the GetDownloadUrl method).
List<ICentralizedFile> GetFiles(PathSearchOption searchOption)
List<ICentralizedFile> GetFiles(string path, PathSearchOption searchOption)
Gets a list of files for the given path matching the defined search option (TopLevelPathOnly, AllPaths). When using the TopLevelPathOnly search option, the file store works like a file system. When using the AllPaths search option, the file store works more like a key-based system.
Note that paths are technically keys, not directories. If I have a file in the path "my.files.are.here", the file can be retrieved from the path "my.files" or "my.fil" when using the AllPaths search option.
List<string> GetPaths()
List<string> GetPaths(string path)
Gets a list of paths within the defined path. Omitting the path will retrieve all top-level paths. Paths are separated using periods (identified by the CentralizedFileSystemProvider.DirectorySeparator property).
void AddPath(string path)
Adds a path.
There is no concept of "parent" folder and "child" folder, instead, paths are treated as keys. So I could add a path named "my.files.are.here" just as easily as incrementally adding "my" then "my.files" then "my.files.are" and finally "my.files.are.here". The CFS will handle either case.
ICentralizedFile AddUpdateFile(string path, string fileName, Stream contentStream)
Adds or updates a file to the file store. If a file with the same path and fileName already exists, it is overwritten. The path does not need to be manually created.
void Delete()
void Delete(string path)
void Delete(string path, string fileName)
Deletes a file or path. If the path is omitted, all paths and files are deleted from the file store.
Implementing New CFS File Storage Providers
New CFS file storage providers can be implemented by inheriting from the CommunityServer.Components.CentralizedFileStorageProvider. A few notes regarding implementing custom CFS storage providers:
-
Are You Sure?
New CFS file storage providers do not need to be implemented to store file in the CFS. Only when you want to store files outside of the file system or Amazon S3 will a new file storage provider be required.
-
Resources
Because the creation of new CFS file storage providers should be rare, I'm not going to list all of the considerations in this article. Instead, I would suggest reviewing the notes and comments in the CentralizedFileSystemProvider class in the CS2008 SDK and the two included implementations for specific information and implementation guidelines for each method.
-
ICentralizedFile
Each CentralizedFileSystemProvider must also implement the ICentralizedFile interface and support reading files as a stream (via the ICentralizedFile.OpenReadStream method) and downloading files via a URL (via the ICentralizedFile.GetDownloadUrl method). All CFS functionality should be fully implemented to support all of the ways a file store can be used.
General CFS Notes
A few general notes about using the CFS in CS2008:
-
Avoid Publishing the ICentralizedFile.GetDownloadUrl()
In general, the CentralizedFileStorageProvider.GetGenericDownloadUrl(ICentralizedFile) method should be used when rendering download links to a file. The URL generated by this method is generic and, when accessed, will determine the current file storage provider for the file and redirect to it. This will allow the storage provider to be changed without having to update all existing links to files (since ICentralizedFile.GetDownloadUrl() would otherwise return a storage-provider-specific URL that would not be valid when the storage provider is changed).
-
Avoid Calling ICentralizedFile.OpenReadStream()
Whenever possible, render a URL and don't read the contents of the file. In most cases, a URL can be rendered to a user instead of reading a file and writing it manually. In some cases, however, this is not possible... such as when resizing an image.
-
When a URL is really a CFS-stored File
When processing URLs, the CentralizedFileStorageProvider class exposes a few static methods that can help to translate a URL into a CFS-stored ICentralizedFile: bool IsCentralizedFile(string url) and ICentralizedFile GetCentralizedFileByUrl(string url). These methods can be used to detect and retrieve CFS files by their URL. Note that these methods only work with URLs generated by the CentralizedFileStorageProvider.GetGenericDownloadUrl method.
Questions? Comments? It was a long article... feel free to ask in the comments.