Skip to content
This repository was archived by the owner on Jun 1, 2024. It is now read-only.
This repository was archived by the owner on Jun 1, 2024. It is now read-only.

Support for data streams Elastic 7.9 #355

Closed
@orjan

Description

@orjan

Does this issue relate to a new feature or an existing bug?

  • New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package.
<PackageReference Include="serilog.sinks.elasticsearch" Version="8.2.0" />

What is the target framework and operating system? See target frameworks & net standard matrix.

  • netCore 3.1.402

Please describe the current behavior?
Data streams is a reliable way to minimize the number of shards in a cluster. Previously you had to setup an ILM, template and a write alias, data streams is more or less the same but endorsed by Elastic an transparent in Kibana and easier to configure.

There a constraint within data stream and it will only support the op_type=create
https://www.elastic.co/guide/en/elasticsearch/reference/master/use-a-data-stream.html#data-streams-bulk-indexing-requests

When trying to index documents from serilog the request looks like this

POST _bulk
{"index":{"_index":"my-data-stream","_type":"_doc"}}
{"@timestamp":"2020-09-11T16:14:17.6858199+02:00","level":"Error","messageTemplate":"Hello data stream from Serilog","message":"Hello data stream from Serilog"}
#! Deprecation: [types removal] Specifying types in bulk requests is deprecated.
{
  "took" : 0,
  "errors" : true,
  "items" : [
    {
      "index" : {
        "_index" : "my-data-stream",
        "_type" : "_doc",
        "_id" : null,
        "status" : 400,
        "error" : {
          "type" : "illegal_argument_exception",
          "reason" : "only write ops with an op_type of create are allowed in data streams"
        }
      }
    }
  ]
}

Please describe the expected behavior?
In order to support data streams we'll need to be able to alter the op_type from index to create then it should work:
https://github.com/serilog/serilog-sinks-elasticsearch/blob/master/src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/ElasticSearchSink.cs#L156-L160

There's also a deprecation warning for the _type removal but that's covered by a separate issue: #345

From an API point of view I can imagine specifying the op_type similar to this?

                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
                {
                    IndexFormat = "my-data-stream",
                    AutoRegisterTemplate = false,
                    BatchOpType = OpType.Create, 

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions