我正在用xarray读取一个相当小的NetCDF文件(5.4MB),并希望将其转换为Pandas数据帧:
import xarray as xr
f = xr.open_dataset('file.nc')
到目前为止,Python使用的内存非常少(约75mb),但是一旦我调用:
^{pr2}$内存使用量呈爆炸式增长(大于12 GB)。有人知道为什么会这样吗?我尝试过使用to_dask_dataframe()
,但这给了我NetCDF文件中某些数据类型的错误。在
我上传了NetCDF文件here,原始数据被随机数替换,因为我无法分发原始数据。在
按照评论中的要求:
In [3]: f
Out[3]:
<xarray.Dataset>
Dimensions: (day_in_time_interval: 3652, nv: 2, time: 175296)
Coordinates:
* time (time) datetime64[ns] 2001-01-01 2001-01-01T00:30:00 ...
Dimensions without coordinates: day_in_time_interval, nv
Data variables:
iso_dataset |S1 ...
product |S1 ...
station_details |S1 ...
date (time) int32 ...
valid_dates (day_in_time_interval) int8 ...
time_bnds (time, nv) float32 ...
C020 (time) float32 ...
C060 (time) float32 ...
C120 (time) float32 ...
C200 (time) float32 ...
Attributes:
institution: Royal Netherlands Meteorological Institute (KNMI)
comment: none
Conventions: CF-1.4
location: CESAR observatory, the Netherlands
file_creation_date_time: 20161130 09:34:56 (UTC)
以及原始文件的ncdump
:
netcdf ecnco2 {
dimensions:
time = UNLIMITED ; // (175296 currently)
nv = 2 ;
day_in_time_interval = 3652 ;
variables:
char iso_dataset ;
iso_dataset:hierarchyLevel = "dataset" ;
iso_dataset:url = "http://www.cesar-database.nl" ;
iso_dataset:protocol = "website" ;
iso_dataset:topic = "climatologyMeteorologyAtmosphere" ;
iso_dataset:westbound_longitude = "4.926" ;
iso_dataset:eastbound_longitude = "4.926" ;
iso_dataset:southbound_latitude = "51.97" ;
iso_dataset:northbound_latitude = "51.97" ;
iso_dataset:datasetDateType = "publication" ;
iso_dataset:code = "28992" ;
iso_dataset:codeSpace = "EPSG" ;
iso_dataset:accessConstraints = "CESAR data policy" ;
iso_dataset:useLimitation = "None" ;
iso_dataset:organisationName_dataset = "Royal Netherlands Meteorological Institute (KNMI)" ;
iso_dataset:email_dataset = "fred.bosveld@knmi.nl" ;
iso_dataset:role_dataset = "Principle Investigator" ;
iso_dataset:organisationName_metadata = "Royal Netherlands Meteorological Institute (KNMI)" ;
iso_dataset:role_metadata = "Principle Investigator" ;
iso_dataset:email_metadata = "fred.bosveld@knmi.nl" ;
iso_dataset:url_metadata = "http://www.knmi.nl/~bosveld" ;
iso_dataset:metadataDateType = "creation" ;
iso_dataset:language = "eng" ;
iso_dataset:metadataStandardName = "ISO-19115" ;
iso_dataset:metadataStandardNameVersion = "Nederlands profiel op ISO 19115 voor geografie, v1.2" ;
char product ;
product:format_version = "netCDF,3.6" ;
product:originator = "Bosveld, F.C., KNMI" ;
product:software_version = "see http://www.knmi.nl/~bosveld -> software -> Mobibase" ;
product:command_line = " ncselect.x ecnco2 a30 [M]cesar,[o]ecnco2 2001,2010 -fecnco2.nc" ;
product:date_start_of_data = "2001-01-01T00:00Z" ;
product:date_end_of_data = "2010-12-31T23:59Z" ;
product:revision_date = "2016-11-30" ;
char station_details ;
station_details:name = "CESAR observatory" ;
station_details:latitude = "51.97" ;
station_details:longitude = "4.926" ;
station_details:elevation = "-0.7" ;
station_details:WMO_id = "06348" ;
station_details:address = "Zijdeweg 1" ;
station_details:postal_code = "3411 MH" ;
station_details:city = "Lopik" ;
station_details:administration_area = "Utrecht" ;
station_details:country = "the Netherlands" ;
float time(time) ;
time:units = "hours since 2001-01-01 00:00:00 0:00" ;
time:long_name = "hours since 2001-01-01 00:00:00 (UTC)" ;
time:standard_name = "time" ;
time:axis = "T" ;
time:bounds = "time_bnds" ;
int date(time) ;
date:long_name = "yyyymmdd" ;
byte valid_dates(day_in_time_interval) ;
valid_dates:comment = "indicates whether any data are included for a particular day: 0=none, 1=data, index runs from date indicated by \"units\" attribute of the time variable" ;
float time_bnds(time, nv) ;
float C020(time) ;
C020:units = "ppm" ;
C020:long_name = "CO2 concentration ECN at 20 m" ;
C020:_FillValue = -9999.f ;
C020:cell_methods = "time: mean" ;
float C060(time) ;
C060:units = "ppm" ;
C060:long_name = "CO2 concentration ECN at 60 m" ;
C060:_FillValue = -9999.f ;
C060:cell_methods = "time: mean" ;
float C120(time) ;
C120:units = "ppm" ;
C120:long_name = "CO2 concentration ECN at 120 m" ;
C120:_FillValue = -9999.f ;
C120:cell_methods = "time: mean" ;
float C200(time) ;
C200:units = "ppm" ;
C200:long_name = "CO2 concentration ECN at 200 m" ;
C200:_FillValue = -9999.f ;
C200:cell_methods = "time: mean" ;
// global attributes:
:institution = "Royal Netherlands Meteorological Institute (KNMI)" ;
:comment = "none" ;
:Conventions = "CF-1.4" ;
:location = "CESAR observatory, the Netherlands" ;
:file_creation_date_time = "20161130 09:34:56 (UTC)" ;
:_Format = "classic" ;
}
这是因为数据集有多个维度,要在一个数据帧中表示所有这些维度,必须进行大量广播。为了说明缩小数据集的大小:
因此,数据集3维最终生成一个3级多重索引,其长度等于我的示例中维度(4*2*2)的乘积。在
您可能希望通过删除一些伪变量/维度来清理数据集。以下行适用于您共享的小数据集或完整数据集:
^{pr2}$相关问题 更多 >
编程相关推荐