{"id":598,"date":"2020-09-23T02:28:24","date_gmt":"2020-09-22T17:28:24","guid":{"rendered":"http:\/\/kei0310.info\/?p=598"},"modified":"2020-09-23T02:32:18","modified_gmt":"2020-09-22T17:32:18","slug":"post-598","status":"publish","type":"post","link":"http:\/\/kei0310.info\/?p=598","title":{"rendered":"python\u3067storage transfer service\u3092\u4f7f\u3063\u3066\u307f\u305f\u3002"},"content":{"rendered":"\n<h2>\u6e96\u5099<\/h2>\n\n\n\n<p>python\u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3068\u304b\u3059\u308b\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u3042\u305f\u308a\u3002<br><a rel=\"noreferrer noopener\" aria-label=\" (\u65b0\u3057\u3044\u30bf\u30d6\u3067\u958b\u304f)\" href=\"http:\/\/kei0310.info\/?p=592\" target=\"_blank\">http:\/\/kei0310.info\/?p=592<\/a><br><\/p>\n\n\n\n<h2>\u30b5\u30f3\u30d7\u30eb\u30b3\u30fc\u30c9\u3092\u4f7f\u3046<\/h2>\n\n\n\n<p>\u3068\u308a\u3042\u3048\u305a\u3001\u30b5\u30f3\u30d7\u30eb\u30b3\u30fc\u30c9\u3092\u4f7f\u3063\u3066\u307f\u308b\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u3042\u305f\u308a\u3092\u4e38\u3054\u3068\u30b3\u30d4\u3063\u3066\u4f7f\u3046\u3002<br><a rel=\"noreferrer noopener\" aria-label=\" (\u65b0\u3057\u3044\u30bf\u30d6\u3067\u958b\u304f)\" href=\"https:\/\/github.com\/GoogleCloudPlatform\/python-docs-samples\/blob\/master\/storage\/transfer_service\/aws_request.py\" target=\"_blank\">https:\/\/github.com\/GoogleCloudPlatform\/python-docs-samples\/blob\/master\/storage\/transfer_service\/aws_request.py<\/a><br><\/p>\n\n\n\n<h2>\u8ee2\u9001\u5143\u306e\u30c7\u30fc\u30bf\u306e\u6e96\u5099<\/h2>\n\n\n\n<p>AWS S3\u306b\u30d0\u30b1\u30c3\u30c8\u3068\u8ee2\u9001\u3059\u308b\u30c7\u30fc\u30bf\u304c\u3042\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3002<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ aws s3 ls s3:\/\/[s3-bucket-name]\/\n2020-09-15 17:01:23          0 test.txt<\/code><\/pre>\n\n\n\n<h2>GCS\u306b\u8ee2\u9001\u5148\u306e\u30d0\u30b1\u30c3\u30c8\u3092\u7528\u610f\u3059\u308b<\/h2>\n\n\n\n<p>gcs\u306b\u8ee2\u9001\u5148\u306e\u30d0\u30b1\u30c3\u30c8\u304c\u3042\u308b\u3053\u3068\u3092\u78ba\u8a8d\u3002<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ gsutil ls gs:\/\/\ngs:\/\/[gcs-bucket-name]\/<\/code><\/pre>\n\n\n\n<h2>\u30b8\u30e7\u30d6\u4f5c\u6210\u7528\u306e\u30b5\u30fc\u30d3\u30b9\u30a2\u30ab\u30a6\u30f3\u30c8\u3092\u7528\u610f\u3059\u308b<\/h2>\n\n\n\n<p>IAM\u304b\u3089\u30b5\u30fc\u30d3\u30b9\u30a2\u30ab\u30a6\u30f3\u30c8\u3092\u65b0\u898f\u4f5c\u6210\u3059\u308b\u3002<br>\u6709\u52b9\u306a\u30b5\u30fc\u30d3\u30b9\u30a2\u30ab\u30a6\u30f3\u30c8\u3067\u3042\u308c\u3070\u3001\u6a29\u9650\u306f\u3064\u3051\u306a\u304f\u3066\u3082\u5927\u4e08\u592b\u305d\u3046\u3002\uff08\u9055\u3046\u304b\u3082\uff09<br><\/p>\n\n\n\n<h2>python\u7528\u30c7\u30d5\u30a9\u30eb\u30c8\u30b5\u30fc\u30d3\u30b9\u30a2\u30ab\u30a6\u30f3\u30c8\u306b\u8a2d\u5b9a<\/h2>\n\n\n\n<p>json\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u3067key\u30d5\u30a1\u30a4\u30eb\u3092\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3057\u3066\u30b5\u30fc\u30d0\u30fc\u4e0a\u306b\u30d5\u30a1\u30a4\u30eb\u3068\u3057\u3066\u914d\u7f6e\u3002<br>\u4ee5\u4e0b\u306e\u30b3\u30de\u30f3\u30c9\u3067python\u304c\u4f7f\u3046\u30b5\u30fc\u30d3\u30b9\u30a2\u30ab\u30a6\u30f3\u30c8\u3068\u3057\u3066\u5b9a\u7fa9\u3059\u308b\u3002<br>\uff08\u5fc5\u8981\u304c\u3042\u308c\u3070\u3001.bash_profile\u3068\u304b\u306b\u3082\u767b\u9332\u3057\u3066\u306d\uff09 <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ export GOOGLE_APPLICATION_CREDENTIALS=\/path\/to\/credential.json<\/code><\/pre>\n\n\n\n<h2>\u8ee2\u9001\u7528\u306e\u30b5\u30fc\u30d3\u30b9\u30a2\u30ab\u30a6\u30f3\u30c8\u3092\u7528\u610f\u3059\u308b<\/h2>\n\n\n\n<p>\u901a\u5e38\u306e\u30b5\u30fc\u30d3\u30b9\u30a2\u30ab\u30a6\u30f3\u30c8\u3068\u5225\u3067\u3001API\u5c02\u7528\u306e\u30b5\u30fc\u30d3\u30b9\u30a2\u30ab\u30a6\u30f3\u30c8\u306e\u8a2d\u5b9a\u3092\u884c\u3046\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u3042\u305f\u308a\u3092\u53c2\u8003\u306b\u3057\u305f\u3002<br><a rel=\"noreferrer noopener\" aria-label=\" (\u65b0\u3057\u3044\u30bf\u30d6\u3067\u958b\u304f)\" href=\"https:\/\/medium.com\/@karnavatpradip12345\/storage-transfer-from-aws-s3-to-gcs-using-api-solution-to-insufficient-permission-error-on-the-ad76c4b89dc0\" target=\"_blank\">https:\/\/medium.com\/@karnavatpradip12345\/storage-transfer-from-aws-s3-to-gcs-using-api-solution-to-insufficient-permission-error-on-the-ad76c4b89dc0<\/a><\/p>\n\n\n\n<p>\u3053\u306e\u30da\u30fc\u30b8\u3092\u958b\u304f\u3002<br><a rel=\"noreferrer noopener\" aria-label=\" (\u65b0\u3057\u3044\u30bf\u30d6\u3067\u958b\u304f)\" href=\"https:\/\/cloud.google.com\/storage-transfer\/docs\/reference\/rest\/v1\/googleServiceAccounts\/get\" target=\"_blank\">https:\/\/cloud.google.com\/storage-transfer\/docs\/reference\/rest\/v1\/googleServiceAccounts\/get<\/a><br><br>\u30da\u30fc\u30b8\u306e\u53f3\u5074\u306b\u5165\u529b\u6b04\u304c\u3042\u308b\u306e\u3067\u3001\u30d7\u30ed\u30b8\u30a7\u30af\u30c8ID\u3092\u5165\u529b\u3057\u3066\u300cEXECUTE\u300d\u3092\u62bc\u3059\u3002<br><br>\u6210\u529f\u3059\u308b\u3068\u3001\u300c200\u300d\u6b04\u306b\u4ee5\u4e0b\u304c\u5165\u529b\u3055\u308c\u308b\u306e\u3067\u3001\u30e1\u30fc\u30eb\u30a2\u30c9\u30ec\u30b9\u90e8\u5206\u3092\u30b3\u30d4\u30fc\u3059\u308b\u3002<br><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"accountEmail\": \"project-xxxxxxxxxx@storage-transfer-service.iam.gserviceaccount.com\"\n}<\/code><\/pre>\n\n\n\n<p>\u30b3\u30d4\u30fc\u3057\u305f\u30e1\u30fc\u30eb\u30a2\u30c9\u30ec\u30b9\u3092\u3001Cloud Storage\u30b3\u30f3\u30bd\u30fc\u30eb\u306e\u30d0\u30b1\u30c3\u30c8\u3092\u9078\u629e\u3057\u305f\u5f8c\u306e\u300c\u6a29\u9650\u300d\u30bf\u30d6\u304b\u3089\u300c+\u8ffd\u52a0\u300d\u3092\u9078\u629e\u3057\u3066\u3001\u300cStorage \u30ec\u30ac\u30b7\u30fc \u30d0\u30b1\u30c3\u30c8\u66f8\u304d\u8fbc\u307f\u300d\u3092\u4ed8\u4e0e\u3059\u308b\u3002<\/p>\n\n\n\n<h2>\u8ee2\u9001\u30b8\u30e7\u30d6\u306e\u5b9f\u884c<\/h2>\n\n\n\n<p>\u8ee2\u9001\u30b8\u30e7\u30d6\u3092\u4ee5\u4e0b\u306e\u30b3\u30de\u30f3\u30c9\u3067\u5b9f\u884c\u3059\u308b<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ python aws_request.py [description] [project-id] [yyyy\/mm\/dd] [HH:MM:SS] [s3_bucket_name] [aws_access_key] [aws_secret_key] [gcs_bucket_name]<\/code><\/pre>\n\n\n\n<p>\u5b9f\u884c\u7d50\u679c\u4f8b<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Returned transferJob: {\n    \"name\": \"transferJobs\/14429711211095668488\",\n    \"description\": \"transfer-test\",\n    \"projectId\": \"xxxxxxxx\",\n    \"transferSpec\": {\n        \"awsS3DataSource\": {\n            \"bucketName\": \"s3_bucket_name\"\n        },\n        \"gcsDataSink\": {\n            \"bucketName\": \"gcs_bucket_name\"\n        }\n    },\n    \"schedule\": {\n        \"scheduleStartDate\": {\n            \"year\": 2020,\n            \"month\": 9,\n            \"day\": 22\n        },\n        \"scheduleEndDate\": {\n            \"year\": 2020,\n            \"month\": 9,\n            \"day\": 22\n        }\n    },\n    \"status\": \"ENABLED\",\n    \"creationTime\": \"2020-09-22T17:18:03.834435704Z\",\n    \"lastModificationTime\": \"2020-09-22T17:18:03.834435704Z\"\n}<\/code><\/pre>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u6e96\u5099 python\u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3068\u304b\u3059\u308b\u3002 \u3053\u306e\u3042\u305f\u308a\u3002http:\/\/kei0310.info\/?p=592 \u30b5\u30f3\u30d7\u30eb\u30b3\u30fc\u30c9\u3092\u4f7f\u3046 \u3068\u308a\u3042\u3048\u305a\u3001\u30b5\u30f3\u30d7\u30eb\u30b3\u30fc\u30c9\u3092\u4f7f\u3063\u3066\u307f\u308b\u3002 \u3053\u306e\u3042\u305f\u308a\u3092\u4e38\u3054\u3068\u30b3\u30d4\u3063\u3066\u4f7f\u3046\u3002https:\/ [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[214,228,212,195,229],"tags":[215,232,231,230,200,234,233,235,21],"_links":{"self":[{"href":"http:\/\/kei0310.info\/index.php?rest_route=\/wp\/v2\/posts\/598"}],"collection":[{"href":"http:\/\/kei0310.info\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/kei0310.info\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/kei0310.info\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/kei0310.info\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=598"}],"version-history":[{"count":5,"href":"http:\/\/kei0310.info\/index.php?rest_route=\/wp\/v2\/posts\/598\/revisions"}],"predecessor-version":[{"id":603,"href":"http:\/\/kei0310.info\/index.php?rest_route=\/wp\/v2\/posts\/598\/revisions\/603"}],"wp:attachment":[{"href":"http:\/\/kei0310.info\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=598"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/kei0310.info\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=598"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/kei0310.info\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=598"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}