Amazon-S3 [SOLVED]: Easy way to created dated subdirectories on AWS S3

Amazon-S3 [SOLVED]: Easy way to created dated subdirectories on AWS S3

Home Forums Amazon Web Services Amazon S3 Amazon-S3 [SOLVED]: Easy way to created dated subdirectories on AWS S3

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #246986

    Cloudy Point
    Keymaster

    QuestionQuestion

    I’m trying to create a web service that is able to store user-upload files in S3. The problem is that we want the files stored in “dated directories”.

    For example, if a user uploads a.txt on 12/1/2017 at 9:15am, the file should look like this in S3:

    https://s3-eu-west-1.amazonaws.com/test-bucket/uploaded/2017/12/1/9/a.txt

    Does S3 have any API to help us achieving this or do we need to hand-craft this solution?

    #246987

    Cloudy Point
    Keymaster

    Accepted AnswerAnswer

    There is no such API in S3. Think of Amazon S3 as a storage service, not an application or database.

    It is the responsibility of your application to store the data in the desired naming format — just like storing data on a disk.

    By the way, your naming format could do with some improvement:

    • Always expand fields to the correct number of digits (use 01 for January rather than 1) so that they sort correctly.
    • Think about your use-case — if you will be scanning documents by year, then the /2017/12/01/09/a.txt naming format makes sense since you can look in the 2017 directory (not that directories really exist in S3). If not, then simply store it as /2017-12-01-09-a.txt.
    • Make it very clear which one is month vs day — the USA is the only country in the world that treats “12/1/2017” as December 1st. The rest of the world reads it as “12 January”. Using the format of 2017-12-01 makes it clear that it is 1-December-2017.
    • What about naming conflicts? Can only one person upload a file with a given name on a given day? How are you going to differentiate between different users uploading a file with the same name?
    • The reality is, the filename is totally irrelevant — your application should use a database to keep track of objects that users
      upload and assign each of them a unique name. When a file is later
      requested, lookup the filename in the database and then provide that
      file. Do not use S3 filenames as a pseudo-database where the name
      conveys particular meaning, otherwise you’ll often have to rename
      files to add more meaning!

      • Directories don’t actually exist in S3 — they are just part of the filename. So, you can create a file in a given directory just by storing it — there is no need to pre-create directories.

    Source: https://stackoverflow.com/questions/47914566/easy-way-to-created-dated-subdirectories-on-aws-s3
    Author: John Rotenstein
    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.