Support for data streams Elastic 7.9 #355
Description
Does this issue relate to a new feature or an existing bug?
- New Feature
What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package.
<PackageReference Include="serilog.sinks.elasticsearch" Version="8.2.0" />
What is the target framework and operating system? See target frameworks & net standard matrix.
- netCore 3.1.402
Please describe the current behavior?
Data streams is a reliable way to minimize the number of shards in a cluster. Previously you had to setup an ILM, template and a write alias, data streams is more or less the same but endorsed by Elastic an transparent in Kibana and easier to configure.
There a constraint within data stream and it will only support the op_type=create
https://www.elastic.co/guide/en/elasticsearch/reference/master/use-a-data-stream.html#data-streams-bulk-indexing-requests
When trying to index documents from serilog the request looks like this
POST _bulk
{"index":{"_index":"my-data-stream","_type":"_doc"}}
{"@timestamp":"2020-09-11T16:14:17.6858199+02:00","level":"Error","messageTemplate":"Hello data stream from Serilog","message":"Hello data stream from Serilog"}
#! Deprecation: [types removal] Specifying types in bulk requests is deprecated.
{
"took" : 0,
"errors" : true,
"items" : [
{
"index" : {
"_index" : "my-data-stream",
"_type" : "_doc",
"_id" : null,
"status" : 400,
"error" : {
"type" : "illegal_argument_exception",
"reason" : "only write ops with an op_type of create are allowed in data streams"
}
}
}
]
}
Please describe the expected behavior?
In order to support data streams we'll need to be able to alter the op_type
from index
to create
then it should work:
https://github.com/serilog/serilog-sinks-elasticsearch/blob/master/src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/ElasticSearchSink.cs#L156-L160
There's also a deprecation warning for the _type
removal but that's covered by a separate issue: #345
From an API point of view I can imagine specifying the op_type similar to this?
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
{
IndexFormat = "my-data-stream",
AutoRegisterTemplate = false,
BatchOpType = OpType.Create,