如何锁定Google Cloud Storage文本文件以实现事务操作

Question

我有一个文本文件（比如叫“X”），它存储在GCS（谷歌云存储）上，是通过GCS客户端库创建和更新的。我使用的是GAE Python（谷歌应用引擎的Python版本）。每当我的网站用户添加一些数据时，我就会在“默认”队列中添加一个任务（taskqueue.Task），这个任务会执行一些操作，包括修改文件（“X”）。

有时候，我在日志中看到以下错误：

E 2014-07-20 03:19:06.238 500 3KB 430ms /t
0.1.0.2 - - [19/Jul/2014:14:49:06 -0700] "POST /t HTTP/1.1" 500 2569 "http://www.myappdomain.com/p" "AppEngine-Google; (+http://code.google.com/appengine)" "www.myappdomain.com" ms=430 cpu_ms=498 cpm_usd=0.000287 queue_name=default task_name=14629523467445182169 instance=00c61b117c48b4db44a58e0d454310843e7848 app_engine_release=1.9.7 trace_id=3db3eb580b76133e90947539c0446910  
   I 03:19:05.813 [class TaskQueueWorker]  work=[sitemap_index_entry]  
   I 03:19:05.813 country_id=[US] country_name=[USA] state_id=[CA] state_name=[California] city_id=[SVL] city_name=[Sunnyvale]  
   I 03:19:05.836 locality_id_old=[-1] locality_id_new=[28]  
   I 03:19:05.879 locality_name_old=[] locality_name_new=[XYZ]  
   I 03:19:05.879 command=[ADD]  
   E 03:19:06.207 File on GCS has changed while reading.  
Traceback (most recent call last):  
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
    rv = self.handle_exception(request, response, e)  
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)  
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)  
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
    return handler.dispatch()  
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)  
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)  
  File "/base/data/home/apps/s~myappdomain/1.377368272328585247/main_v3.py", line 15259, in post
    gcs_file = gcs.open (index_filename, mode='r')  
  File "/base/data/home/apps/s~myappdomain/1.377368272328585247/cloudstorage/cloudstorage_api.py", line 94, in open
    buffer_size=read_buffer_size)  
  File "/base/data/home/apps/s~myappdomain/1.377368272328585247/cloudstorage/storage_api.py", line 220, in __init__
    check_response_closure()  
  File "/base/data/home/apps/s~myappdomain/1.377368272328585247/cloudstorage/storage_api.py", line 448, in _checker
    self._check_etag(resp_headers.get('etag'))  
  File "/base/data/home/apps/s~myappdomain/1.377368272328585247/cloudstorage/storage_api.py", line 476, in _check_etag
    raise ValueError('File on GCS has changed while reading.')  
ValueError: File on GCS has changed while reading.  
   I 03:19:06.235 Saved; key: __appstats__:045800, part: 144 bytes, full: 74513 bytes, overhead: 0.002 + 0.004; link: http://www.myappdomain.com/_ah/stats/details?time=1405806545812

我怀疑是因为多个任务同时被触发，试图打开并更新文件（“X”），这导致了上面的异常。请给我建议一个方法来锁定对这个文件的访问，这样一次只有一个任务可以修改它（类似于事务处理）。

非常感谢你的帮助和指导。

更新
另一种防止上述问题的方法可能是修改以下队列的queue.yaml参数之一：

bucket_size

或者

max_concurrent_requests

但我不确定该修改哪一个。

transaction management concurrency control cloud computing task queue data integrity google cloud storage file locking queue configuration

如何锁定Google Cloud Storage文本文件以实现事务操作

2 个回答

撰写回答