Cython与Hadoop编译错误..有解决方案吗？

Question

我正在尝试运行Hadoopy，但是在OS X上遇到了编译错误：

ImportError: Building module failed: ["CompileError: command 'llvm-gcc-4.2' failed with exit status 1\n"

我的$PATH里有/Developer/usr/bin，而且我在OS X Lion 10.7上运行的是最新版本的XCode。Cython是通过easy_install安装的。

完整输出：

>>> import pyximport; pyximport.install()
>>> import hadoopy
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c:236:22: error: getdelim.h: No such file or directory
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c:236:22: error: getdelim.h: No such file or directory
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c: In function ‘__pyx_f_7hadoopy_5_main_11HadoopyTask_read_offset_value_text’:
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c:4399: warning: implicit conversion shortens 64-bit value into a 32-bit value
lipo: can't open input file: /var/folders/8b/n0j5pn_13qn_x8p2v4f848zh0000gn/T//ccC8x2Ex.out (No such file or directory)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/hadoopy/__init__.py", line 22, in <module>
    from _main import run, print_doc_quit
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyximport.py", line 335, in load_module
    self.pyxbuild_dir)
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyximport.py", line 183, in load_module
    so_path = build_module(name, pyxfilename, pyxbuild_dir)
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyximport.py", line 167, in build_module
    reload_support=pyxargs.reload_support)
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyxbuild.py", line 85, in pyx_to_dll
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/Cython/Distutils/build_ext.py", line 135, in run
    _build_ext.build_ext.run(self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/Cython/Distutils/build_ext.py", line 143, in build_extensions
    self.build_extension(ext)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 499, in build_extension
    depends=ext.depends)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 624, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/unixccompiler.py", line 180, in _compile
    raise CompileError, msg
ImportError: Building module failed: ["CompileError: command 'llvm-gcc-4.2' failed with exit status 1\n"

相关问题：

• 在Os X上安装Cython时出错
• 在OS X上安装Pycurl时出错
• Cython: 找不到符号 _PyCFunction_Check
• 如何在Lion中设置PyDev/Eclipse？
• 在OS X Snow Leopard上安装PyCairo或Py2Cairo
• 在OS X 10.6上安装rpy2时出错：无法执行gcc-4.0
• 在OS X 10.6上安装scrapy的问题
• PySide/Qt中Jpeg插件未加载

Answer 1

对我来说，这是一个安装的问题，我之前通过以下步骤解决了这个问题：

• 首先，运行命令：pip install argparse

• 然后，使用这个命令下载代码：git clone https://github.com/bwhite/hadoopy.git

• 接着，进入下载的文件夹：cd hadoopy

• 最后，运行这个命令来安装：python setup.py install

Answer 2

与其使用pyximport，不如直接在当前目录下构建扩展模块，可以用命令 python setup.py build_ext --inplace。如果你想在开发过程中直接使用，可以用 python setup.py develop，或者通过 python setup.py install 进行常规安装。对于大多数包来说，运行setup是很重要的，这样可以正确配置编译环境、构建和安装过程。

---

如果你在用Cython来加速代码（比如科学计算），那么pyximport对于个人脚本来说是不错的选择。不过，有时候你可能需要引入其他库，并且从多个源构建。在这种情况下，你可以使用一个pyxbld文件来设置源文件和包含的目录。例如，如果你有一个叫foo.pyx的文件，你可以在同一目录下放一个名为foo.pyxbld的文件，里面写上构建指令。比如：

#foo.pyxbld
def make_ext(modname, pyxfilename):
    from distutils.extension import Extension
    return Extension(name = modname,
                     sources=[pyxfilename, 'bar.c'],
                     include_dirs=['/myinclude'] )
def make_setup_args():
    return dict(script_args=["--compiler=mingw32"])

现在使用foo就简单多了，像下面这样：

import pyximport
pyximport.install()
import foo

假设没有编译错误，第一次导入模块时可能会有点延迟，但之后的导入就会很快，因为它会在你家目录或个人目录下的.pyxbld子目录中查找已构建的扩展。