The S3 virtual file system is used to handle documents stored in an Amazon S3 cloud as if they were local files in an Adelia cloud application. It is based on an implementation of a Java NIO file system implementing all the basic functions of a file system.
The file system is compatible with Minio (https://min.io/) in particular, which allows access to private cloud-type storage to be declared.
Declaring an S3 file system
The S3 virtual file system is implemented by the Java class "com.hardis.wagon.s3.vfs.WagonS3FileSystem". It is declared in the usual way in the "wagon.xml" file.
NB: the server root can be specified as the root path (rootPath="/"). In this case, only the buckets belonging explicitly to the user will be visible and will be processed as directories.
Example of declaring a VFS S3 on the "data" bucket:
|
All the standard VFS configuration parameters are applicable. The connection parameters specific to the VFS S3 are as follows:
endpoint | Server address in "host:port" form. For example: "myminio:9000" |
region | Region S3 |
accessKey | Application access key or user ID. |
secretKey | Secret key of the application or user's password. |
| |
connectionTimeout | Connection timeout in milliseconds, optional. |
protocol | Connection protocol (http or https). By default: http. |
trustStore | Used to specify an alternative truststore for secure connections. By default: not specified, uses the JVM truststore. |
trustStoreType | Truststore type. By default: jks. |
trustStorePassword | Truststore password. By default: changeit. |
keyStore | Used to specify an alternative keystore for secure connections. By default: not specified, uses the JVM keystore. |
keyStoreType | Keystore type. By default: jks. |
keyStorePassword | Keystore password. By default: changeit. |
keyPassword | Key password. By default: changeit. |
| |
overrideWritePermission | The VFS uses the S3 ACLs to determine whether an object is accessible in write mode. In certain situations (Minio when you are not the bucket, owner, insufficient rights), the ACLs cannot be read.
In this case, this parameter specifies that, by default, we can assume that the objects are accessible in write mode. This parameter is optional and must only be used if necessary.
Possible values: true, false, false by default. |
S3 file system limitations
The VFS S3 transparently emulates a "normal" file system. To maximize interoperability, this file system uses the same separator as the native file system (i.e. "\" in Windows, "/" in Unix platforms).
The VFS complies with the POSIX standard for hidden files: a file with a name starting with a "." cannot be seen by file search functions ("VaToolBxFindFirst", "VaToolBxFindNext", file explorer), but may be accessed directly by the management functions (create, delete, read and edit).
There are, however, certain limitations we need to be aware of.
Managing access to files in read and write mode
Files are accessed via read or write operations by transparently downloading the file content when opening it when required, and when opening in write mode, by uploading it to the server when closing. All the intermediary operations are performed on a copy of the file in the machine's temporary directory.
This involves the following limitations:
Concurrent access cannot be managed and files cannot be locked: two people can open the same file at the same time and, if it is edited, the last person to close the file will define its content.
Upload times may reduce performance for large files.
File closing may fail due to uploading and the fact that in certain cases it is impossible to correctly determine the write rights on a file (which is rarely the case in a conventional file system).
It is therefore necessary to pay attention to the result of the "VaToolBxCloseFile" method call for files open for editing.
The VFS does not implement multi-upload, which limits the size of writable files to 5GB, and does not support GLACIER storage.
Managing file access rights
VFS S3 uses S3 ACLs to determine the write permission on an object or bucket. However, this method is only partially implemented by "Minio" - ACL reading does not work if you do not own the object, and S3 may not be configured to allow ACL reading.
To authorize write access in this case, the VFS defines a parameter to force the default write permission if ACL reading is not supported or configured (overrideWritePermission, see above). If this parameter is incorrectly configured, the VFS may authorize the opening of a file in write mode but file closing fails with an "access denied" error.
ACL modification is not supported. "VaToolBxSetReadOnly" and "VaToolBxSetWritable" functions are not supported by VFS S3. ↑ Top of page
Temporary file management
Although VFS S3 authorizes the creation of temporary files, it is not recommended. Cloud storage must be seen as a long-term storage support, temporary files must be created in a local directory, preferably in the user's or system's temporary directory.
It is also not advisable to create "log" type files (which remain open for a long period of time and are only closed at the end of the session) in VFS S3.
Managing directories
In reality, S3 is a flat file system that does not have the notion of directory. Directories only exist virtually when they contain files. If all the files present in a directory are deleted (in reality all the objects with a name prefixed by the directory name), the directory itself no longer exists.
For example, an S3 object called "data/logs/wagon.log" will be interpreted as the "wagon.log" file in the "logs" sub-directory of the "data" directory.
To simplify and make directory management transparent, VFS S3 automatically creates a hidden file called "directory_name/.virtual_folder" when you create a directory. In addition, VFS will consider a directory as "empty" (so can be deleted by "VaToolBxRemoveDirectory") when it ONLY contains this ".virtual_folder" file.
When the VFS is created from the root, it displays the buckets belonging to the connected user (accessKey and secretKey parameters of the VFS configuration) as directories. In this case:
A directory can be created at the root (creation of a bucket), but this directory must comply with the S3 naming standard (lowercase alphanumeric characters and numbers, no spaces or underline characters).
The Adelia file manager in particular cannot create a folder at the root of an S3 as the "New folder" name does not comply with the naming standard.
A file cannot be created at the root: a file must be in a bucket.
A bucket which is accessible to the user but not owned by them will not appear in the file explorer or in the directory lists. The objects of this bucket can however be accessed if the full path is provided.