Aws S3 Change To Infrequent Access
A new feature that could be helpful is AWS S3 Auto Tiering. In short, any objects not accessed within 30 days are moved to infrequent access class. You can have a lifecycle rule modify the storage class to intelligent tiering on upload - it may take a few hours. To move objects to Glacier storage class you still need a lifecycle rule. Mar 19, 2019 - Amazon S3 Standard Infrequent Access. You can save objects directly in Glacier or set a Lifecycle Policy on your S3 bucket to automatically.
My understanding is since Restic v0.8 there is a local cache of the indexes stored on S3, so the indexes probably aren't going to be downloaded and used regularly.
Lifecycle Policy
You can set a lifecycle policy to move data to S3-IA class that applies only to data files as per this example.
If you have a bucket that contains a restic backup you should use a prefix
data/
If you have a bucket with a subfolder called 'PhotosBackup' that has a restic backup inside the prefix would be
PhotosBackup/data/
With this information it's fairly simple to have S3 change data files to IA class, leaving indexes and everything else standard class.
Storage Costs
Assumptions (I don't know Restic internals at all):
- Files on S3 are immutable. Confirmed here. Immutable until they're potentially deleted anyway.
- We're not restoring data regularly. If you are then then there will be extra costs.
- Prices are for the US-East region 15/7/2018. Prices charge fairly regularly.
S3 standard currently costs $0.023 per GB, S3-IA costs $0.0125 per GB. Transitions to IA class cost $0.01 per 1,000 requests. You can go to single region IA but the saving isn't great (25%) and you lose durability.
If you have 1TB of data that's going to be around 100,000 files. S3 standard storage costs $23.56 per month, IA $12.80. Transitioning 100K objects from standard to IA costs $1. So even in your first month you come out ahead, and in a year you save around $128 by using S3-IA class for 1TB of data.
Retrieval Costs
There's no retrieval costs for S3 standard class, just bandwidth charges. S3-IA charges 1c/GB to retrieve data.
Cost to restore 1TB:
- S3 standard - $0.09 / GB = $90 for 1TB
- S3 IS= ($0.01 * 1000) + S3 standard cost = $100 for 1TB
Using S3 Glacier Class
Glacier is cold storage. It's designed for long term storage of years or decades, rarely accessed. Accessing an object takes 3-5 hours at standard rates, plus there are fast and slow options, but none are near real time.
S3's Glacier storage class is $0.004c/GB, which is about 1/6 of S3 standard and 1/3 of S3-IA class. What the pricing page doesn't mention (but the FAQ does) is that you effectively have a three month minimum storage charge. Any object deleted in less than three months is charged for the full three months.
If your data is largely static, such as a family photo archive, it could make sense to transition data to S3 Glacier storage class. I'd probably move to IA class after some time, then maybe leave it 3 months before transitioning to S3-Glacier.
Amazon S3 Standard Infrequent Access
If the 1TB of data mentioned above was in Glacier class it would cost $4.10 per month, plus a $5 fee for moving data into the Glacier class. That's a heck of a lot less than S3 standard.
Retrieval using S3-Glacier storage class will be somewhat manual I think. Retrieval is triggered using S3, you get a temporary copy which lives for 24 hours by default, which I assumed is stored in the original location (I haven't found this in the docs). Assuming it's in the original location you could then use Restic to retrieve your data. If you need a significant fraction of your data you might be better off downloading all the data and doing a local restore.
Aws S3 Bandwidth
Comparison with B2 Pricing
1996 bmw 840ci. B2 storage is $0.005c/GB. It's reliability is similar to S3-One Zone IA, which is $0.01c/GB, so B2 is half the price of S3. Most S3 classes are higher reliability than B2, with data for most S3 classes stored in three AZs (data centers). I believe B2 has data in a single Data Center.
Aws S3 Change To Infrequent Access Storage
AWS bandwidth charges are very high at $0.09c/GB, whereas B2 is $0.01/GB. Given AWS has much higher scale than B2 this must be a significant profit center for them.