使用python将刮取的数据直接保存到azure blob存储

Question

我的需要如下：

我需要使用azure函数从网页中刮取数据，并将刮取输出直接以.csv或.parquet格式保存到azure blob存储中

请参见示例代码：

response = requests.get('example_url')
        json_response = response.json()
        name = json_response.get('name', 'N/A')
        with open('warsaw.txt', 'w') as f:
            f.write(name)

如何使用Blob Service Client将f直接写入azure blob存储

Answer 1

您可以使用Azure Storage Blobs client library for Python

有关如何将文件上载到Azure存储Blob的示例：

from azure.storage.blob import BlobClient

blob = BlobClient.from_connection_string(conn_str="<connection_string>", container_name="my_container", blob_name="my_blob")

with open("./warsaw.txt", "rb") as data:
    blob.upload_blob(data)

Answer 2

I need to scrape the data from the webpage using azure functions, and save the scraping output directly to the azure blob storage as a .csv or .parquet.

我认为您可以使用“附加”方法来实现您的要求：

from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, BlobType
try:
    connect_str = "DefaultEndpointsProtocol=https;AccountName=1123bowman;AccountKey=xxxxxx;EndpointSuffix=core.windows.net"
    container_name = "test"
    blob_name = "test.txt"
    data1 = "\n1,2,3"
    data = str.encode(data1)
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
    print("length of data is "+str(len(data)))
    blob_client.append_block(data,length=len(data))
except:
    connect_str = "DefaultEndpointsProtocol=https;AccountName=1123bowman;AccountKey=xxxxxx;EndpointSuffix=core.windows.net"
    container_name = "test"
    blob_name = "test.txt"
    data1 = "test1,test2,test3"
    data = str.encode(data1)
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
    print("length of data is "+str(len(data)))
    blob_client.upload_blob(data,blob_type=BlobType.AppendBlob)

通过上述方法，您不必每次都存储整个blob