尝试将文件从云存储读取到存储时,获取google.auth.exceptions.RefreshError

2024-04-19 08:21:25 发布

您现在位置:Python中文网/ 问答频道 /正文

我试图在本地将云存储文件读入Pandas数据帧,然后使用Python将其加载到一个大查询表中,但出现以下错误:

google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{\n "error": "invalid_grant",\n "error_description": "Bad Request"\n}')

我的代码是:

import pandas as pd
from google.cloud import storage
from google.oauth2 import service_account
import gcsfs

scopes=["https://www.googleapis.com/auth/bigquery","https://www.googleapis.com/auth/cloud-platform"]

file='<<filename>>.json'
credentials = service_account.Credentials.from_service_account_file(file, scopes=scopes)

def run(file):
    client = storage.Client.from_service_account_json(file)
    bucket = client.bucket('my_bucket')
    blobs = bucket.list_blobs()
    list_temp_raw = []
    for file in blobs:
        filename = file.name
        temp = pd.read_csv('gs://<<>my_bucket>>/' + filename, encoding='utf-8')
        list_temp_raw.append(temp)
    df = pd.concat(list_temp_raw, encoding='utf-8')
    df.to_gbq('<<dataset>>.<<table>>',if_exists='append',credentials=credentials)

run(file)

temp = pd.read_csv('gs://<<>my_bucket>>/' + filename, encoding='utf-8')导致了错误:

Traceback (most recent call last):
  File "test-load-bq.py", line 23, in <module>
    run(file)
  File "test-load-bq.py", line 18, in run
    temp = pd.read_csv('gs://project-test/' + filename, encoding='utf-8')
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/pandas/io/parsers.py", line 431, in _read
    filepath_or_buffer, encoding, compression
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/pandas/io/common.py", line 192, in get_filepath_or_buffer
    filepath_or_buffer, encoding=encoding, compression=compression, mode=mode
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/pandas/io/gcs.py", line 17, in get_filepath_or_buffer
    filepath_or_buffer = fs.open(filepath_or_buffer, mode)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/spec.py", line 936, in open
    **kwargs
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 1287, in _open
    **kwargs,
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 1412, in __init__
    **kwargs,
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/spec.py", line 1257, in __init__
    self.details = fs.info(path)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/asyn.py", line 121, in wrapper
    return maybe_sync(func, self, *args, **kwargs)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/asyn.py", line 100, in maybe_sync
    return sync(loop, func, *args, **kwargs)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/asyn.py", line 71, in sync
    raise exc.with_traceback(tb)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/fsspec/asyn.py", line 55, in f
    result[0] = await future
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 825, in _info
    return await self._get_object(path)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 607, in _get_object
    res = await self._call("GET", "b/{}/o/{}", bucket, key, json_out=True)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 503, in _call
    self.maybe_refresh()
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/gcsfs/core.py", line 404, in maybe_refresh
    self.credentials.refresh(req)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/google/oauth2/credentials.py", line 198, in refresh
    self._scopes,
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/google/oauth2/_client.py", line 248, in refresh_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/Users/willcharles/Documents/project/venv/lib/python3.6/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{\n  "error": "invalid_grant",\n  "error_description": "Bad Request"\n}')

我已为我的服务帐户指定以下角色: 仓库管理及;BigQuery管理员

知道错误发生的原因吗


Tags: inpyprojectvenvlibpackagesgoogleline