flink time and watermark
流处理中时间本质上就是一个普通的递增字段(long型,自1970年算起的微秒数),不一定真的表示时间。
watermark只是应对乱序的办法之一,大多是启发式的,在延迟和完整性之间抉择。(如果没有延迟,就不够完整;如果有延迟,极端情况就是批处理,当然完整性足够高)
org.apache.flink.streaming.api.watermark
Class Watermark
java.lang.Object
org.apache.flink.streaming.runtime.streamrecord.StreamElement
org.apache.flink.streaming.api.watermark.Watermark
@PublicEvolving
public final class Watermark extends StreamElement
A Watermark tells operators that no elements with a timestamp older or equal to the watermark timestamp should arrive at the operator. Watermarks are emitted at the sources and propagate through the operators of the topology. Operators must themselves emit watermarks to downstream operators using Output.emitWatermark(Watermark). Operators that do not internally buffer elements can always forward the watermark that they receive. Operators that buffer elements, such as window operators, must forward a watermark after emission of elements that is triggered by the arriving watermark.
In some cases a watermark is only a heuristic and operators should be able to deal with late elements. They can either discard those or update the result and emit updates/retractions to downstream operations.
When a source closes it will emit a final watermark with timestamp Long.MAX_VALUE. When an operator receives this it will know that no more input will be arriving in the future.
Modifier and Type Field and Description
static Watermark MAX_WATERMARK
The watermark that signifies end-of-event-time.
reference:
https://www.bilibili.com/video/av53193640/
https://ci.apache.org/projects/flink/flink-docs-release-1.9/api/java/
免责声明:
① 本站未注明“稿件来源”的信息均来自网络整理。其文字、图片和音视频稿件的所属权归原作者所有。本站收集整理出于非商业性的教育和科研之目的,并不意味着本站赞同其观点或证实其内容的真实性。仅作为临时的测试数据,供内部测试之用。本站并未授权任何人以任何方式主动获取本站任何信息。
② 本站未注明“稿件来源”的临时测试数据将在测试完成后最终做删除处理。有问题或投稿请发送至: 邮箱/279061341@qq.com QQ/279061341