Edgewall Software
Modify

Opened 5 years ago

Last modified 16 months ago

#11675 assigned enhancement

IAttachmentStorage interface for pluggable attachment storage

Reported by: Ryan J Ollos Owned by: Ryan J Ollos
Priority: normal Milestone: 1.5.1
Component: general Version:
Severity: normal Keywords:
Cc: Jun Omae Branch:
Release Notes:
API Changes:

Description (last modified by Ryan J Ollos)

The interface would allow attachments to be saved to external storage, such as DropBox. This has been discussed for implementation in bloodhound:#486. Related to #11339.

Attachments (0)

Change History (23)

comment:1 by Jun Omae, 5 years ago

Cc: Jun Omae added

comment:2 by Ryan J Ollos, 5 years ago

Owner: set to Ryan J Ollos
Status: newassigned

comment:3 by Ryan J Ollos, 5 years ago

Milestone: 1.1.31.1.4

comment:4 by Ryan J Ollos, 5 years ago

Milestone: 1.1.41.1.5

Narrowing focus for 1.1.4.

comment:5 by Ryan J Ollos, 5 years ago

Milestone: 1.1.51.2

comment:6 by Ryan J Ollos, 5 years ago

Milestone: 1.21.1.6

Milestone renamed

comment:7 by Ryan J Ollos, 5 years ago

Milestone: 1.1.6next-dev-1.1.x
Owner: Ryan J Ollos removed
Status: assignednew

comment:8 by Ryan J Ollos, 4 years ago

Related idea that may be possible with additional interfaces is to allow VersionControlSystems for attachment storage.

comment:9 by Ryan J Ollos, 4 years ago

Milestone: next-dev-1.1.xnext-dev-1.3.x

Narrowing focus for milestone:1.2. Please move ticket to milestone:1.2 if you intend to fix it.

comment:10 by Ryan J Ollos, 3 years ago

I did some work on factoring out an IAttachmentStorage interface: log:rjollos.git:t11675_iattachmentstorage_interface. Next, I'll look at interfacing with a remote storage service. Apache libcloud looks interesting, so I may write a plugin that uses it. I'm posting the changes early in case anyone wants to give feedback.

I'm unsure if/how XSendFile can be used with a file served by an external resource, but will investigate that while implementing plugins that implement IAttachmentStorage for various services.

comment:11 by Peter Suter, 3 years ago

This sounds great. The IAttachmentStorage interface looks nice.

I assume for remote storage services this would mean all attachment uploads and downloads will still be routed through the Trac server? (Not directly between the browser and the remote storage.)

(This would presumably solve one of the main problems with running Trac on Heroku. There's still #11339, but apparently that can be worked around somehow.)

in reply to:  11 comment:12 by Ryan J Ollos, 3 years ago

Replying to Peter Suter:

I assume for remote storage services this would mean all attachment uploads and downloads will still be routed through the Trac server? (Not directly between the browser and the remote storage.)

I'm unsure if it's possible to route directly between browser and remote storage, but I plan to research that. It seems like a widely useful optimization given the prevalence of services like Amazon S3.

(This would presumably solve one of the main problems with running Trac on Heroku. There's still #11339, but apparently that can be worked around somehow.)

Trac on Heroku is the main target I have in mind and I plan to look at #11339 after this ticket is complete. It would be nice to have a Deploy in Heroku button on WikiStart, so that users can have a Trac instance running in just a few clicks.

I'll also look at implementing an IAttachmentStorage interface for a document database, such as MongoDB.

comment:13 by Peter Suter, 3 years ago

I have also wondered for a while if a deploy button would be a good idea. (At least one alternative exists in Deploy on OpenShift, and I guess BitNami might also be possible.)

For Amazon S3 I've used a redirect to a boto3.generate_presigned_url for downloading files directly in a similar use-case. This seemed surprisingly simple. Direct uploading was a bit more complicated (client-side logic and a new server-side endpoint for boto3.generate_presigned_post).

(But for Trac it would probably be largely acceptable to route through the server for many installations where attachments are relatively small / infrequent.)

Last edited 3 years ago by Peter Suter (previous) (diff)

comment:14 by Ryan J Ollos, 2 years ago

Description: modified (diff)
Summary: FileSystem component and abstract interfaceIAttachmentStorage interface for pluggable attachment storage

comment:15 by Ryan J Ollos, 2 years ago

Rebased and added mock attachment storage to EnvironmentStub: [5026314b4/rjollos.git]. I'll rebase again after #12870 is committed and then continue work on this feature.

comment:16 by Ryan J Ollos, 2 years ago

Rebased again in rjollos.git:t11675_iattachmentstorage_interface.2. Rather than making the change in comment:76:ticket:11901 I'm going to look into finishing this ticket for 1.4. I need to add documentation and develop some example implementations of IAttachmentStorage.

comment:17 by Ryan J Ollos, 2 years ago

Milestone: next-dev-1.3.x1.3.3
Owner: set to Ryan J Ollos
Status: newassigned

comment:18 by Jun Omae, 2 years ago

I think we should document file-like object is returned from IAttachmentStorage.open(). In addition, the object should be closable if it has close method, or closable by with statement.

Version 0, edited 2 years ago by Jun Omae (next)

comment:19 by Ryan J Ollos, 19 months ago

Milestone: 1.3.31.3.4

in reply to:  18 comment:20 by Ryan J Ollos, 17 months ago

Replying to Jun Omae:

I think we should document file-like object is returned from IAttachmentStorage.open(). In addition, the object should be closable if it has close method, and closable by with statement.

I'll look into addressing those items.

Latest rebased changes in log:rjollos.git:t11675_iattachmentstorage_interface.4.

in reply to:  18 comment:21 by Ryan J Ollos, 16 months ago

Replying to Jun Omae:

I think we should document file-like object is returned from IAttachmentStorage.open(). In addition, the object should be closable if it has close method, and closable by with statement.

Does this documentation address the need that you see?: rjollos.git/trac/attachment.py@d460b5d82#L131. Let me know if I'm missing anything.

comment:22 by Ryan J Ollos, 16 months ago

I implemented S3AttachmentStorage using the boto3 library.

Here's an S3 bucket policy for user TracTest and bucket trac-test (<account id> removed):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account id>:user/TracTest"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::trac-test"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account id>:user/TracTest"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::trac-test/*"
        }
    ]
}

I implemented the comment:13 suggestion in [0ac27a216/rjollos.git].

I'm reconsidering the IAttachmentStorage's open method. BotoCore has a StreamingBody method that we might be able to use via the Object.get method if open is changed to return an iterable.

comment:23 by Ryan J Ollos, 16 months ago

Milestone: 1.3.41.5.1

Modify Ticket

Change Properties
Set your email in Preferences
Action
as assigned The owner will remain Ryan J Ollos.
The ticket will be disowned. Next status will be 'new'.
as The resolution will be set. Next status will be 'closed'.
to as assigned The owner will be changed from Ryan J Ollos to the specified user.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.