从数据框架生成变更日志

Question

我每天都在对一些页面进行爬网，我想跟踪每次爬网之间的变化。基本上每天我都从页面中得到我需要的东西，并把它写在历史表格中。然后，对于每个URL，我从历史数据生成一个pandas数据帧。我做到了以下几点：

                          from            to
crawl_id                                    
20190609 price              50           100
20190613 price             100           140
         vdp_url  www.url1.com  www.url2.com
20190614 vdp_url  www.url2.com  www.url1.com
20190616 vdp_url  www.url1.com  www.url3.com

我需要生成这样的东西：

[{"date": "20190609", "from": 50, "to": 100, "field": "price"}, {"date": "20190613", "from": 100, "to": 140, "field": "price"},{"date": "20190613", "from": "www.url1.com", "to": "www.url2.com", "field": "vdp_url"}, {"date": "20190614", "from": "www.url2.com", "to": "www.url1.com", "field": "vdp_url"}, {"date": "20190616", "from": "www.url1.com", "to": "www.url3.com", "field": "vdp_url"}]

这是我用来生成上述数据帧的代码：

histories_df = [{'crawl_id': '20190606', 'vdp_url': 'www.url1.com', 'price': None},
{'crawl_id': '20190607', 'vdp_url': 'www.url1.com', 'price': None},
{'crawl_id': '20190609', 'vdp_url': 'www.url1.com', 'price': 50},
{'crawl_id': '20190613', 'vdp_url': 'www.url1.com', 'price': 100},
{'crawl_id': '20190614', 'vdp_url': 'www.url2.com', 'price': 140},
{'crawl_id': '20190615', 'vdp_url': 'www.url1.com', 'price': None},
{'crawl_id': '20190616', 'vdp_url': 'www.url1.com', 'price': 140},
{'crawl_id': '20190617', 'vdp_url': 'www.url3.com', 'price': 140}]
histories_df = pd.DataFrame(histories_df)
trimmed_histories = histories_df.set_index('crawl_id')
histories_df_prev = trimmed_histories.shift(-1)
diff_bool = trimmed_histories.where(trimmed_histories.values != histories_df_prev.values).notna().stack()
difference = pd.concat([trimmed_histories.stack()[diff_bool], histories_df_prev.stack()[diff_bool]], axis=1).dropna()
difference.columns=['from', 'to']

我已经试着用普通的for、iterrows、indexes、groupby和我发现的任何东西来实现这个目标好几个小时了，但是运气不好

谢谢

Answer 1

嗯，用to_dict

difference.rename_axis(['date','field']).reset_index().to_dict('r')
Out[128]: 
[{'date': '20190609', 'field': 'price', 'from': 50.0, 'to': 100.0},
 {'date': '20190613', 'field': 'price', 'from': 100.0, 'to': 140.0},
 {'date': '20190613',
  'field': 'vdp_url',
  'from': 'www.url1.com',
  'to': 'www.url2.com'},
 {'date': '20190614',
  'field': 'vdp_url',
  'from': 'www.url2.com',
  'to': 'www.url1.com'},
 {'date': '20190616',
  'field': 'vdp_url',
  'from': 'www.url1.com',
  'to': 'www.url3.com'}]