IAttachmentStorage interface for pluggable attachment storage

[Ryan J Ollos]

The interface would allow attachments to be saved to external storage, such as ​DropBox. This has been discussed for implementation in bloodhound:#486. Related to #11339.

[Ryan J Ollos]

Narrowing focus for 1.1.4.

[Ryan J Ollos]

Milestone renamed

[Ryan J Ollos]

Related idea that may be possible with additional interfaces is to allow VersionControlSystems for attachment storage.

[Ryan J Ollos]

Narrowing focus for milestone:1.2. Please move ticket to milestone:1.2 if you intend to fix it.

[Ryan J Ollos]

I did some work on factoring out an IAttachmentStorage interface: log:rjollos.git:t11675_iattachmentstorage_interface. Next, I'll look at interfacing with a remote storage service. ​Apache libcloud looks interesting, so I may write a plugin that uses it. I'm posting the changes early in case anyone wants to give feedback.

I'm unsure if/how XSendFile can be used with a file served by an external resource, but will investigate that while implementing plugins that implement IAttachmentStorage for various services.

[Peter Suter]

This sounds great. The IAttachmentStorage interface looks nice.

I assume for remote storage services this would mean all attachment uploads and downloads will still be routed through the Trac server? (Not directly between the browser and the remote storage.)

(This would presumably solve one of the main problems with ​running Trac on Heroku. There's still #11339, but apparently that can be worked around somehow.)

[Ryan J Ollos]

Replying to Peter Suter:

I assume for remote storage services this would mean all attachment uploads and downloads will still be routed through the Trac server? (Not directly between the browser and the remote storage.)

I'm unsure if it's possible to route directly between browser and remote storage, but I plan to research that. It seems like a widely useful optimization given the prevalence of services like Amazon S3.

(This would presumably solve one of the main problems with ​running Trac on Heroku. There's still #11339, but apparently that can be worked around somehow.)

Trac on Heroku is the main target I have in mind and I plan to look at #11339 after this ticket is complete. It would be nice to have a ​Deploy in Heroku button on WikiStart, so that users can have a Trac instance running in just a few clicks.

I'll also look at implementing an IAttachmentStorage interface for a document database, such as MongoDB.

[Peter Suter]

I have also wondered for a while if a deploy button would be a good idea. (At least one alternative exists in ​Deploy on OpenShift, and I guess BitNami might also be possible.)

For Amazon S3 I've used a redirect to a ​boto3.generate_presigned_url for downloading files directly in a similar use-case. This seemed surprisingly simple. Direct uploading was a bit more complicated (client-side logic and a new server-side endpoint for ​boto3.generate_presigned_post).

(But for Trac it would probably be largely acceptable to route through the server for many installations where attachments are relatively small / infrequent.)

[Ryan J Ollos]

Rebased and added mock attachment storage to EnvironmentStub: [5026314b4/rjollos.git]. I'll rebase again after #12870 is committed and then continue work on this feature.

[Ryan J Ollos]

Rebased again in rjollos.git:t11675_iattachmentstorage_interface.2. Rather than making the change in comment:76:ticket:11901 I'm going to look into finishing this ticket for 1.4. I need to add documentation and develop some example implementations of IAttachmentStorage.

[Jun Omae]

I think we should document file-like object is returned from IAttachmentStorage.open(). In addition, the object should be closable if it has close method, and closable by with statement.

[Ryan J Ollos]

Replying to Jun Omae:

I think we should document file-like object is returned from IAttachmentStorage.open(). In addition, the object should be closable if it has close method, and closable by with statement.

I'll look into addressing those items.

Latest rebased changes in log:rjollos.git:t11675_iattachmentstorage_interface.4.

[Ryan J Ollos]

Replying to Jun Omae:

I think we should document file-like object is returned from IAttachmentStorage.open(). In addition, the object should be closable if it has close method, and closable by with statement.

Does this documentation address the need that you see?: rjollos.git/trac/attachment.py@d460b5d82#L131​. Let me know if I'm missing anything.

[Ryan J Ollos]

I implemented S3AttachmentStorage using the boto3 library.

Here's an S3 bucket policy for user TracTest and bucket trac-test (<account id> removed):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account id>:user/TracTest"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::trac-test"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account id>:user/TracTest"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::trac-test/*"
        }
    ]
}

I implemented the comment:13 suggestion in [0ac27a216/rjollos.git].

I'm reconsidering the IAttachmentStorage's open method. BotoCore has a ​StreamingBody method that we might be able to use via the ​Object.get method if open is changed to return an iterable.

[Ryan J Ollos]

Milestone renamed