在Spark中将大型文本/pgn文件转换为JSON

[Event "Rated Classical game"] [Site "https://lichess.org/j1dkb5dw"] [White "BFG9k"] [Black "mamalak"] [Result "1-0"] [UTCDate "2012.12.31"] [UTCTime "23:01:03"] [WhiteElo "1639"] [BlackElo "1403"] [WhiteRatingDiff "+5"] [BlackRatingDiff "-8"] [ECO "C00"] [Opening "French Defense: Normal Variation"] [TimeControl "600+8"] [Termination "Normal"] 1. e4 e6 2. d4 b6 3. a3 Bb7 4. Nc3 Nh6 5. Bxh6 gxh6 6. Be2 Qg5 7. Bg4 h5 8. Nf3 Qg6 9. Nh4 Qg5 10. Bxh5 Qxh4 11. Qf3 Kd8 12. Qxf7 Nc6 13. Qe8# 1-0

ncalls tottime percall cumtime percall filename:lineno(function) 1 26.828 26.828 395.848 395.848 /Users/danieljones/Documents – Daniel’s iMac/GitHub/ST446Project/ParsePGN.py:11(parse_pgn) 20000 0.798 0.000 289.203 0.014 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/frame.py:7614(append) 20000 0.098 0.000 199.489 0.010 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/reshape/concat.py:70(concat) 20000 0.480 0.000 126.548 0.006 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/reshape/concat.py:295(__init__) 100002 0.212 0.000 122.178 0.001 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/generic.py:5199(_protect_consolidate) 80002 0.076 0.000 122.177 0.002 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/generic.py:5210(_consolidate_inplace) 40000 0.079 0.000 122.063 0.003 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/generic.py:5218(_consolidate) 80002 0.170 0.000 121.830 0.002 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/generic.py:5213(f) 100001 0.223 0.000 99.829 0.001 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/internals/managers.py:986(_consolidate_inplace) 59999 0.451 0.000 96.718 0.002 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/internals/managers.py:1898(_consolidate) 80002 0.138 0.000 96.599 0.001 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/internals/managers.py:970(consolidate) 79999 52.602 0.001 91.913 0.001 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/internals/managers.py:1915(_merge_blocks) 20000 7.432 0.000 79.741 0.004 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/chess/pgn.py:1323(read_game) 20000 0.361 0.000 72.843 0.004 /Users/danieljones/opt/anaconda3/envs/LSE/lib/python3.6/site-packages/pandas/core/reshape/concat.py:456(get_result)

def parse_pgn(pgn): games = [] i = 0 edges_df = pd.DataFrame(columns=["Event", "Round", "WhitePlayer", "BlackPlayer", "Result", "BlackElo", "Opening", "TimeControl", "Date", "Time", "WhiteElo"]) while i < 20000: first_game = chess.pgn.read_game(pgn) if first_game is not None: Event = first_game.headers["Event"] Round = first_game.headers["Round"] White_player = first_game.headers["White"] Black_player = first_game.headers["Black"] Result = first_game.headers["Result"] # Add condition to split this if Result == "1-0": Result = White_player elif Result == "0-0": Result = "Draw" else: Result = Black_player BlackELO = first_game.headers["BlackElo"] Opening = first_game.headers["Opening"] TimeControl = first_game.headers["TimeControl"] UTCDate = first_game.headers["UTCDate"] UTCTime = first_game.headers["UTCTime"] WhiteELO = first_game.headers["WhiteElo"] edges_df = edges_df.append({"Event": Event, "Round": Round, "WhitePlayer": White_player, "BlackPlayer": Black_player, "Result": Result, "BlackElo": BlackELO, "Opening": Opening, "TimeControl": TimeControl, "Date": UTCDate, "Time": UTCTime, "White": WhiteELO, }, ignore_index=True) games.append(first_game) i += 1 else: pass return edges_df

1条回答

网友

1楼 · 发布于 2024-06-16 13:40:54

不要在循环中.append到pandas.DataFrame如果您想要有较短的运行时间，您可以阅读更多关于这个here的内容。您可以首先将dict存储在iterable中，然后从中创建pandas.DataFrame。我将使用collections.deque（来自collections内置模块），因为它设计用于高速运动.append，让我们比较一下这些不同的方式

import collections
import pandas as pd
def func1():
    df = pd.DataFrame(columns=['x','y','z'])
    for i in range(1000):
        df = df.append({'x':i,'y':i*10,'z':i*100}, ignore_index=True)
    return df
def func2():
    store = collections.deque()
    for i in range(1000):
        store.append({'x':i,'y':i*10,'z':i*100})
    df = pd.DataFrame(store, columns=['x','y','z'])
    return df

这些函数产生相等的pandas.DataFrame，我使用内置模块timeit按如下方式比较它们

import timeit
print(timeit.timeit('func1()',number=10,globals={'func1':func1}))
print(timeit.timeit('func2()',number=10,globals={'func2':func2}))

结果如下

18.370871236000312
0.02604325699940091

这要快500多倍。当然，您的里程数可能会有所不同，但我建议您尝试一下这种优化

相关问题更多 >

编程相关推荐

热门问题

热门文章