删除查询时Django ValidationError

Question

我正在写一个Django命令从我的应用程序中删除超过x天的数据。你知道吗

使用以下方法进行过滤：

qs = Data.objects.filter(date_created__lte=timezone.now()-timedelta(days=days_del))

其中days_del为整数，date_created为DateTimeField。你知道吗

当尝试打印此查询集或对其调用.delete()时，结果是JSONDecodeError和ValidationError。我真的不明白为什么会发生这种情况，也不知道如何防止它在这种情况下尝试解码JSON文件。你知道吗

注意，我使用的是jsonfieldpypi包，数据模型有一个JSONField。你知道吗

有可能是某些数据被暂停并导致了问题（请参阅回溯），是否有方法忽略验证并继续删除？你知道吗

Traceback (most recent call last):
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/jsonfield/fields.py", line 83, in pre_init
    return json.loads(value, **self.load_kwargs)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 464 (char 463)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/webapps/myproj/server/mirrors/management/commands/data_cleanup.py", line 39, in handle
    qs.delete()
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/query.py", line 616, in delete
    collector.collect(del_query)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/deletion.py", line 191, in collect
    reverse_dependency=reverse_dependency)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/deletion.py", line 89, in add
    if not objs:
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/query.py", line 254, in __bool__
    self._fetch_all()
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/query.py", line 1118, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/query.py", line 63, in __iter__
    obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/base.py", line 583, in from_db
    new = cls(*values)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/base.py", line 502, in __init__
    _setattr(self, field.attname, val)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/jsonfield/subclassing.py", line 43, in __set__
    obj.__dict__[self.field.name] = self.field.pre_init(value, obj)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/jsonfield/fields.py", line 85, in pre_init
    raise ValidationError(_("Enter valid JSON"))
django.core.exceptions.ValidationError: ['Enter valid JSON']

我同时删除了很多数据，也许有更好的方法来处理。无论如何，修复旧的过时数据在这里不是一个选项。你知道吗

谢谢

---

以下是命令文件：

from django.core.management.base import BaseCommand, CommandError
from oauth2_provider.models import Application
from django.utils import timezone

import pytz
from datetime import timedelta

from confluence_core.models import Data



class Command(BaseCommand):
    help = 'Delete data older than given days'

    def add_arguments(self, parser):
        parser.add_argument("-d", "--days", type=int, dest='days', required=True, help="Days limit")
        parser.add_argument("-c", "--confirm", action='store_true', dest='confirm', default=False, required=False, help="Confirm before deletion")

    def handle(self, *args, **options):

        days_del = options['days']
        do_delete = False

        qs = Data.objects.filter(date_created__lte=timezone.now()-timedelta(days=days_del))

        if qs.count() > 0:

            if options['confirm']:
                print(f"{qs.count():,} data entries will be deleted.")
                ret = input("Confirm ? (y/n)\n")
                if ret in ['y', 'Y', 'yes']:
                    do_delete = True
            else:
                do_delete = True

            if do_delete is True:
                print(f"Deleting {qs.count():,} data entries...")
                qs.delete()
            else:
                print("Not deleting anything.")

        else:
            print("No data to delete.")

        print("Done.")

Answer 1

假设删除时ORM强制加载查询集，cf:

File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/deletion.py", line 89, in add
if not objs:

我能想到的唯一解决方法（不需要forking或monkeypatching）是首先更新整个queryset，使jsonfield设置为有效的值，即：

qs.update(name_of_the_field={})
qs.delete()

但这并不能避免其他数据不一致的问题，所以真正的解决方案显然是清理整个数据集。你知道吗