Nanoseconds being truncated in asof

[schwallie]

match_time = get_info.index.asof(trade_time)
match_price = get_info[match_time]['Bid Price']

The actual index value:
<Timestamp: 2013-04-10 10:22:01.696815975-0500 CDT, tz=US/Central>

The error returned:
KeyError: u'no item named 2013-04-10 10:22:01.696815-05:00'

It seems to me that it is cutting off the last digits of the timestamp in this example.

[jreback]

can you show pandas version, platform, and get_info?

[schwallie]

pandas version = .10
platform = linux

get_info is a huge huge dataframe that is indexed by a timeseries. I think we should be fine without?

FYI if I use

match_price = get_info[get_info.index[0]]

I get the same problem and error

[jreback]

of course, just want to see a reproducible sample (you can put random data)

[jreback]

related to #3060

0.11rc1

In [14]: s = Series([pd.Timestamp('20130101')]).values.view('i8')[0]

In [15]: r = pd.DatetimeIndex([ s + 50 + i for i in range(100) ])

In [16]: x = Series(randn(100),index=r)

In [17]: x.asof(x.index[0])
Out[17]: -1.7275857659530389

In [20]: x.index.asof(x.index[0])
Out[20]: <Timestamp: 2013-01-01 00:00:00.000000050>

The errors

In [21]: x['2013-01-01 00:00:00.000000050']
KeyError: '2013-01-01 00:00:00.000000050'
In [23]: x[pd.Timestamp('2013-01-01 00:00:00.000000050')]
KeyError: 1356998400000000000

[wuan]

when writing

x[Timestamp(np.datetime64('2013-01-01 00:00:00.000000050+0000', 'ns'))]

the index access should work with #3060 now. The major problem here is, that timestrings are parsed with python datetime when creating a Timestamp object. This ignores nanosecond values.

It may break a lot of things, but wouldnt it be better to use np.datetime64 to parse timestamp strings?

[jreback]

http://comments.gmane.org/gmane.comp.python.pydata/688

the issue is dateutil which truncates

can't use numpy parser as its broken in < 1.7
it's possible to try it first and then fallback to dateutil
might be a perf hit
also possible to detect via some regexes common formats that we know numpy can parse

[wuan]

Maybe the only (performant) way to improve the situation is to reimplement the dateutil parser which is pure python at the moment.

[jreback]

it's possible that we do parse a few common formats in cython/c and fallback to parse
but this would be some work

[jreback]

IIRC this is fixed by some recent work in 0.15.0. If still not working, pls open a new issue.