azurepython存储块blob存储正在耗尽所有内存

Question

我写了一个Python脚本来自动构建一个azurevm并从KVM上传到Azure，我面临一个我无法修复的问题。 一旦虚拟机构建完成，我就试图使用azurepython模块将磁盘上传到Azure，问题是脚本实际上正在吞噬所有可用的RAM。我试过几种编码方法，结果总是一样的。在

   block_blob_service = BlockBlobService(vars.az_storage_acc_name, vars.az_sto_key)
    blob = open(args.pool_path + args.name + "-az"+'.vhd', 'r')
    print "Upload {} to Azure Blob service".format(args.name +"-az"+'.vhd')
    block_blob_service.create_blob_from_stream(vars.az_cnt, args.name +"-az"+'.vhd', blob)

我也尝试了以下方法：

^{pr2}$

运气不好，每次创建blob时都会失败，因为没有可用的RAM。在

你有没有线索可以让我做到这一点？在

Answer 1

感谢您的意见。在

我不明白的是，到底有什么区别

block_blob_service.create_blob_from_stream

以及

^{pr2}$

如果它想把所有的东西都放在内存里？在

Answer 2

这将需要将整个流保存在内存中，除非您的计算机中有max RAM size，否则此代码将无法工作，并在某个时刻给您systemoutofemory异常。在

我建议你把流分块上传，而不是一次性地写。在

这里有一个函数可以将流分块上传

def _upload_blob_chunks(blob_service, container_name, blob_name,
                        blob_size, block_size, stream, max_connections,
                        progress_callback, validate_content, lease_id, uploader_class,
                        maxsize_condition=None, if_modified_since=None, if_unmodified_since=None, if_match=None,
                        if_none_match=None, timeout=None,
                        content_encryption_key=None, initialization_vector=None, resource_properties=None):
    encryptor, padder = _get_blob_encryptor_and_padder(content_encryption_key, initialization_vector,
                                                       uploader_class is not _PageBlobChunkUploader)

    uploader = uploader_class(
        blob_service,
        container_name,
        blob_name,
        blob_size,
        block_size,
        stream,
        max_connections > 1,
        progress_callback,
        validate_content,
        lease_id,
        timeout,
        encryptor,
        padder
    )

    uploader.maxsize_condition = maxsize_condition

    # Access conditions do not work with parallelism
    if max_connections > 1:
        uploader.if_match = uploader.if_none_match = uploader.if_modified_since = uploader.if_unmodified_since = None
    else:
        uploader.if_match = if_match
        uploader.if_none_match = if_none_match
        uploader.if_modified_since = if_modified_since
        uploader.if_unmodified_since = if_unmodified_since

    if progress_callback is not None:
        progress_callback(0, blob_size)

    if max_connections > 1:
        import concurrent.futures
        from threading import BoundedSemaphore

        '''
        Ensures we bound the chunking so we only buffer and submit 'max_connections' amount of work items to the executor.
        This is necessary as the executor queue will keep accepting submitted work items, which results in buffering all the blocks if
        the max_connections + 1 ensures the next chunk is already buffered and ready for when the worker thread is available.
        '''
        chunk_throttler = BoundedSemaphore(max_connections + 1)

        executor = concurrent.futures.ThreadPoolExecutor(max_connections)
        futures = []
        running_futures = []

        # Check for exceptions and fail fast.
        for chunk in uploader.get_chunk_streams():
            for f in running_futures:
                if f.done():
                    if f.exception():
                        raise f.exception()
                    else:
                        running_futures.remove(f)

            chunk_throttler.acquire()
            future = executor.submit(uploader.process_chunk, chunk)

            # Calls callback upon completion (even if the callback was added after the Future task is done).
            future.add_done_callback(lambda x: chunk_throttler.release())
            futures.append(future)
            running_futures.append(future)

        # result() will wait until completion and also raise any exceptions that may have been set.
        range_ids = [f.result() for f in futures]
    else:
        range_ids = [uploader.process_chunk(result) for result in uploader.get_chunk_streams()]

    if resource_properties:
        resource_properties.last_modified = uploader.last_modified
        resource_properties.etag = uploader.etag

    return range_ids

作为参考，你可以浏览下面的线程

https://github.com/Azure/azure-storage-python/blob/master/azure-storage-blob/azure/storage/blob/_upload_chunking.py

同样，对于同一类型的请求也有类似的线程

how to transfer file to azure blob storage in chunks without writing to file using python

或者，您可以使用powershell将VHD上载到vm存储帐户，如下所示

^{pr2}$

这是相同的参考

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/upload-generalized-managed

希望有帮助。在