Data not being saved to my elastic index

[megakid]

Hi there,

I can't quite figure out my issue here. Perhaps I'm using serilog and this sink in ways that I am not meant to. My issue is that I am seeing this sink bookmarking the durable file successfully but not seeing the data in my index!

I have a single public static IConnectionPool SharedElasticConnection { get; } property that I use to instantiate all my elastic clients (inc usages of this sink)

I am instantiating multiple seriloggers - one per type of object I want to save to elastic - each with this elastic sink configured in pretty much the same way. They are held by my generic class public class ElasticPersistorService<TDocument>

var elasticOptions = new ElasticsearchSinkOptions(_settings.ConnectionPool)
{
    IndexFormat = "MyIndex-{0:yyyy.MM.dd}",
    TypeName = typeof(TDocument).Name,
    AutoRegisterTemplate = true,
    CustomFormatter = new ElasticsearchJsonFormatter(inlineFields: true, closingDelimiter: string.Empty), // we do this to make renderMessage: false (this is a textual representation of our data object)
    CustomDurableFormatter = new ElasticsearchJsonFormatter(inlineFields: true, closingDelimiter: string.Empty), // we do this to make renderMessage: false (this is a textual representation of our data object)
    elasticOptions.BufferBaseFilename = $"{thisLogFolder}\\{typeof(TDocument).Name}-buffer"
};
_seriLogger = new LoggerConfiguration()
    .WriteTo.Elasticsearch(elasticOptions)
    .CreateLogger()
    .ForContext<TDocument>();

I use this _seriLogger property in one place:

public void SendToElastic(TDocument d)
{
    _seriLogger.Information("{@" + typeof(TDocument).Name + "}", d);
}

I want these loggers to place documents formatted by serilog and this sink, into a single index (but each with a different type name) but they aren't appearing...

For example today, using the above code, I have MyType-buffer-20160309.json sized at 74,832KB with the latest MyType-buffer.bookmark containing:

76647204:::C:\Serilog\MyType-buffer-20160309.json`.  

It is my understanding that this means that the sink has determined a successful Index operation for the majority of the file. However, I only 1 or 2 documents for each type in my ElasticSearch instance. I am not using LogStash - just straight to Elastic via HTTP.

The strange thing is that when I create a new index, say MyIndexNew-{0:yyyy.MM.dd}, it all works nicely for a while then stops again.

Any ideas?

[megakid]

Maybe it is related to #25 or some threading issue I haven't thought about.

[megakid]

Doh - I just realised my durable files were being appended without line breaks.

The fix was changing:

CustomFormatter = new ElasticsearchJsonFormatter(inlineFields: true, closingDelimiter: string.Empty), // we do this to make renderMessage: false (this is a textual representation of our data object)
CustomDurableFormatter = new ElasticsearchJsonFormatter(inlineFields: true, closingDelimiter: string.Empty), // we do this to make renderMessage: false (this is a textual representation of our data object)

to

CustomFormatter = new ElasticsearchJsonFormatter(inlineFields: true), // we do this to make renderMessage: false (this is a textual representation of our data object)
CustomDurableFormatter = new ElasticsearchJsonFormatter(inlineFields: true), // we do this to make renderMessage: false (this is a textual representation of our data object)

[megakid]

I did create a PR #39 before I found the cause. Might be useful later on.

[davidATceedo]

please note also that index format can't be upper case, and it must be lowercase (i.e. IndexFormat = "myindex-{0:yyyy.MM.dd}" ...)

[killnine]

@davidATceedo Wow man, you dropped some knowledge on me there ;) Thanks for the tip. I was wondering why I couldn't get my data to write out and it was because I had upper casing. Am I missing some documentation that says as much?