将CGImage转换为python图像（pil/opencv）

Question

我想在我的屏幕上做一些模式识别，并将使用Quartz/PyObjc库来获取屏幕截图。在

我得到的截图是CGImage。我想用openCV库在其中搜索模式，但似乎找不到如何将数据转换为openCV可读的。在

所以我想做的是：

#get screenshot and reference pattern
img = getScreenshot() # returns CGImage instance, custom function, using Quartz
reference = cv2.imread('ref/reference_start.png') #get the reference pattern

#search for the pattern using the opencv library
result = cv2.matchTemplate(screen, reference, cv2.TM_CCOEFF_NORMED)

#this is what I need
minVal,maxVal,minLoc,maxLoc = cv2.minMaxLoc(result)

我不知道怎么做，也无法通过谷歌找到信息。在

Answer 1

下面的代码将截图并保存到文件中。要将其读入PIL，只需使用标准Image(path)。如果将区域的大小保持在较小的范围内，这段代码的速度惊人地快。对于800x800像素的区域，在我的i7上，每次拍摄所需时间少于50ms。对于双监视器设置（2880x1800+2560x1440）的完全分辨率，每次拍摄大约需要1.9秒。在

来源：https://github.com/troq/flappy-bird-player/blob/master/screenshot.py

import Quartz
import LaunchServices
from Cocoa import NSURL
import Quartz.CoreGraphics as CG

def screenshot(path, region=None):
    """saves screenshot of given region to path
    :path: string path to save to
    :region: tuple of (x, y, width, height)
    :returns: nothing
    """
    if region is None:
        region = CG.CGRectInfinite

    # Create screenshot as CGImage
    image = CG.CGWindowListCreateImage(
        region,
        CG.kCGWindowListOptionOnScreenOnly,
        CG.kCGNullWindowID,
        CG.kCGWindowImageDefault)

    dpi = 72 # FIXME: Should query this from somewhere, e.g for retina displays

    url = NSURL.fileURLWithPath_(path)

    dest = Quartz.CGImageDestinationCreateWithURL(
        url,
        LaunchServices.kUTTypePNG, # file type
        1, # 1 image in file
        None
        )

    properties = {
        Quartz.kCGImagePropertyDPIWidth: dpi,
        Quartz.kCGImagePropertyDPIHeight: dpi,
        }

    # Add the image to the destination, characterizing the image with
    # the properties dictionary.
    Quartz.CGImageDestinationAddImage(dest, image, properties)

    # When all the images (only 1 in this example) are added to the destination,
    # finalize the CGImageDestination object.
    Quartz.CGImageDestinationFinalize(dest)


if __name__ == '__main__':
    # Capture full screen
    screenshot("testscreenshot_full.png")

    # Capture region (100x100 box from top-left)
    region = CG.CGRectMake(0, 0, 100, 100)
    screenshot("testscreenshot_partial.png", region=region)

Answer 2

我一直在玩这个，但我需要一点更高的性能，所以保存到一个文件，然后再从它读取是有点太慢了。经过一番寻找和摆弄之后，我终于想出了一个办法：

#get_pixels returns a image reference from CG.CGWindowListCreateImage
imageRef = self.get_pixels()
pixeldata = CG.CGDataProviderCopyData(CG.CGImageGetDataProvider(imageRef))
image = Image.frombuffer("RGBA", (self.width, self.height), pixeldata, "raw", "RGBA", self.stride, 1)
#Color correction from BGRA to RGBA
b, g, r, a = image.split()
image = Image.merge("RGBA", (r, g, b, a))

另外请注意，由于我的图像不是标准大小（必须填充），它有一些奇怪的行为，所以我不得不调整缓冲区的步幅，如果你是从标准屏幕宽度拍摄全屏屏幕截图，你可以以0为步长，它会自动计算它。在

现在，您可以将PIL格式转换为numpy数组，以便在OpenCV中更容易使用：

^{pr2}$

Answer 3

为了补充Arqu的答案，你可能会发现使用它更快np.frombuffer如果您的最终目标是使用opencv或numpy，那么不要首先创建PIL图像，因为np.frombuffer花费的时间与图像.frombuffer, 但是节省了从图像转换到numpy数组的步骤（在我的机器上大约需要100ms（其他的都需要~50ms））。在

import Quartz.CoreGraphics as CG
from PIL import Image 
import time
import numpy as np

ct = time.time()
region = CG.CGRectInfinite

# Create screenshot as CGImage
image = CG.CGWindowListCreateImage(
    region,
    CG.kCGWindowListOptionOnScreenOnly,
    CG.kCGNullWindowID,
    CG.kCGWindowImageDefault)

width = CG.CGImageGetWidth(image)
height = CG.CGImageGetHeight(image)
bytesperrow = CG.CGImageGetBytesPerRow(image)

pixeldata = CG.CGDataProviderCopyData(CG.CGImageGetDataProvider(image))
image = np.frombuffer(pixeldata, dtype=np.uint8)
image = image.reshape((height, bytesperrow//4, 4))
image = image[:,:width,:]

print('elapsed:', time.time() - ct)