|
76 | 76 | "Gotchas: There are a couple of considerations to be aware of when working with data on the cloud:\n", |
77 | 77 | "\n", |
78 | 78 | "- Pay-as-you-go - Most cloud providers use pay-as-you-go pricing, where you only pay for the storage and services that you use. This can potentially reduce costs, especially upfront costs (e.g., you never need to buy a hard drive). However, **you may want to provide indefinite access or you may forget about data in storage, in both cases you may end up continuing to pay for data storage indefinitely**.\n", |
79 | | - "- Time and cost of bringing data to your computer - Hosting the data on the cloud naturally means it's no longer already near your computer's processing resources. Transporting data from the cloud to your computer is expensive, since most cloud providers charge for any data leaving their network, and slow, since the data needs to travel large distances. The primary solution for this is \"data-proximate computing\" which involves running your code on computing resources in the same cloud location as your data. For example, I commonly use NASA data products that are hosted on AWS servers in the 'us-west-2' region, which corresponds to Oregon in the figure above. Following the \"data-proximate computing\" paradigm, I use AWS compute resources that are also in Oregon when working with those data, rather than downloading data to use the computing resources on my laptop in North Carolina. In addition to \"data-proximate computing\", there are many other ways to make working with data on the cloud cheaper and easier. Let's take a look!" |
| 79 | + "- Time and cost of bringing data to your computer - Hosting the data on the cloud naturally means it's no longer already near your computer's processing resources. Transporting data from the cloud to your computer is expensive, since most cloud providers charge for any data leaving their network, and slow, since the data needs to travel large distances. \n", |
| 80 | + "\n", |
| 81 | + "The primary solution for the second bullet, \"time and cost bringing data to your computer\", is \"data-proximate computing\" which involves running your code on computing resources in the same cloud location as your data. For example, I commonly use NASA data products that are hosted on AWS servers in the 'us-west-2' region, which corresponds to Oregon in the figure above. Following the \"data-proximate computing\" paradigm, I use AWS compute resources that are also in Oregon when working with those data, rather than downloading data to use the computing resources on my laptop in North Carolina. In addition to \"data-proximate computing\", there are many other ways to make working with data on the cloud cheaper and easier. Let's take a look!" |
80 | 82 | ] |
81 | 83 | }, |
82 | 84 | { |
|
2996 | 2998 | "\n", |
2997 | 2999 | "- [Cloud-Optimized Geospatial Formats Guide](https://guide.cloudnativegeo.org/)\n", |
2998 | 3000 | "- [Xarray Tutorial - Zarr in Cloud Object Storage](https://tutorial.xarray.dev/intermediate/remote_data/cmip6-cloud.html)\n", |
2999 | | - "- [Xarray Tutorial - Access Patterns to Remote Data with fsspec](https://tutorial.xarray.dev/intermediate/remote_data/cmip6-cloud.html)\n" |
| 3001 | + "- [Xarray Tutorial - Access Patterns to Remote Data with fsspec](https://tutorial.xarray.dev/intermediate/remote_data/cmip6-cloud.html)\n", |
| 3002 | + "- [ICESAT-2 Cloud Computing Tutorial](https://icesat-2-2024.hackweek.io/tutorials/cloud-computing/00-goals-and-outline.html)\n" |
3000 | 3003 | ] |
| 3004 | + }, |
| 3005 | + { |
| 3006 | + "cell_type": "markdown", |
| 3007 | + "id": "7e59c7aa", |
| 3008 | + "metadata": {}, |
| 3009 | + "source": [] |
3001 | 3010 | } |
3002 | 3011 | ], |
3003 | 3012 | "metadata": { |
3004 | 3013 | "kernelspec": { |
3005 | | - "display_name": "04-data-in-the-cloud", |
| 3014 | + "display_name": "default", |
3006 | 3015 | "language": "python", |
3007 | 3016 | "name": "python3" |
3008 | 3017 | }, |
|
3016 | 3025 | "name": "python", |
3017 | 3026 | "nbconvert_exporter": "python", |
3018 | 3027 | "pygments_lexer": "ipython3", |
3019 | | - "version": "3.12.0" |
| 3028 | + "version": "3.14.2" |
3020 | 3029 | } |
3021 | 3030 | }, |
3022 | 3031 | "nbformat": 4, |
|
0 commit comments