Django应用中的线程处理

Question

我正在尝试创建一个我称之为“scrobbler”的东西。这个任务是从一个队列中读取一个Delicious用户，获取他们所有的书签，然后把这些书签放入书签队列中。接下来，应该有一些东西去处理这个队列，进行一些解析，然后把数据存储到数据库里。

这显然需要使用多线程，因为大部分时间都是在等待Delicious的响应，然后再等待书签网站的响应，以及通过一些API进行处理。如果所有的操作都在等待，那就太傻了。

不过，我在使用多线程时遇到了一些麻烦，老是出现奇怪的错误，比如数据库表没有定义。任何帮助都非常感谢 :)

这是相关的代码：

# relevant model #
class Bookmark(models.Model):
 account = models.ForeignKey( Delicious )
 url = models.CharField( max_length=4096 )
 tags = models.TextField()
 hash = models.CharField( max_length=32 )
 meta = models.CharField( max_length=32 )

# bookmark queue reading #
def scrobble_bookmark(account):
 try:
  bookmark = Bookmark.objects.all()[0]
 except Bookmark.DoesNotExist:
  return False

 bookmark.delete()

 tags = bookmark.tags.split(' ')
 user = bookmark.account.user

 for concept in Concepts.extract( bookmark.url ):
  for tag in tags:
   Concepts.relate( user, concept['name'], tag )

 return True

def scrobble_bookmarks(account):
 semaphore = Semaphore(10)
 for i in xrange(Bookmark.objects.count()):
  thread = Bookmark_scrobble(account, semaphore)
  thread.start()

class Bookmark_scrobble(Thread):
 def __init__(self, account, semaphore):
  Thread.__init__(self)
  self.account = account
  self.semaphore = semaphore

 def run(self):
  self.semaphore.acquire()
  try:
   scrobble_bookmark(self.account)
  finally:
   self.semaphore.release()

这是我遇到的错误：

Exception in thread Thread-65:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()
  File     "/home/swizec/Documents/trees/bookmarklet_server/../bookmarklet_server/Scrobbler/Scrobbler.py", line 60, in run
    scrobble_bookmark(self.account)
  File     "/home/swizec/Documents/trees/bookmarklet_server/../bookmarklet_server/Scrobbler/Scrobbler.py", line 28, in scrobble_bookmark
    bookmark = Bookmark.objects.all()[0]
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 152, in __getitem__
    return list(qs)[0]
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 76, in __len__
self._result_cache.extend(list(self._iter))
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 231, in iterator
    for row in self.query.results_iter():
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 281, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 2373, in execute_sql
cursor.execute(sql, params)
  File "/usr/local/lib/python2.6/dist-packages/django/db/backends/sqlite3/base.py", line 193, in execute
    return Database.Cursor.execute(self, query, params)
OperationalError: no such table: Scrobbler_bookmark

PS：所有其他依赖于同一张表的测试都通过了。

数据库 django 错误处理多线程数据解析 api 队列处理用户书签

Django应用中的线程处理

4 个回答

撰写回答