我的编程空间,编程开发者的网络收藏夹
学习永远不晚

flink time and watermark

短信预约 信息系统项目管理师 报名、考试、查分时间动态提醒
省份

北京

  • 北京
  • 上海
  • 天津
  • 重庆
  • 河北
  • 山东
  • 辽宁
  • 黑龙江
  • 吉林
  • 甘肃
  • 青海
  • 河南
  • 江苏
  • 湖北
  • 湖南
  • 江西
  • 浙江
  • 广东
  • 云南
  • 福建
  • 海南
  • 山西
  • 四川
  • 陕西
  • 贵州
  • 安徽
  • 广西
  • 内蒙
  • 西藏
  • 新疆
  • 宁夏
  • 兵团
手机号立即预约

请填写图片验证码后获取短信验证码

看不清楚,换张图片

免费获取短信验证码

flink time and watermark

流处理中时间本质上就是一个普通的递增字段(long型,自1970年算起的微秒数),不一定真的表示时间。

watermark只是应对乱序的办法之一,大多是启发式的,在延迟和完整性之间抉择。(如果没有延迟,就不够完整;如果有延迟,极端情况就是批处理,当然完整性足够高)

 

org.apache.flink.streaming.api.watermark
Class Watermark
  java.lang.Object
  org.apache.flink.streaming.runtime.streamrecord.StreamElement
  org.apache.flink.streaming.api.watermark.Watermark

@PublicEvolving
public final class Watermark extends StreamElement

  A Watermark tells operators that no elements with a timestamp older or equal to the watermark timestamp should arrive at the operator. Watermarks are emitted at the sources and propagate through the operators of the topology. Operators must themselves emit watermarks to downstream operators using Output.emitWatermark(Watermark). Operators that do not internally buffer elements can always forward the watermark that they receive. Operators that buffer elements, such as window operators, must forward a watermark after emission of elements that is triggered by the arriving watermark.
  In some cases a watermark is only a heuristic and operators should be able to deal with late elements. They can either discard those or update the result and emit updates/retractions to downstream operations.
  When a source closes it will emit a final watermark with timestamp Long.MAX_VALUE. When an operator receives this it will know that no more input will be arriving in the future.

Modifier and Type     Field and Description
static Watermark      MAX_WATERMARK
             The watermark that signifies end-of-event-time.

reference:

https://www.bilibili.com/video/av53193640/

https://ci.apache.org/projects/flink/flink-docs-release-1.9/api/java/

免责声明:

① 本站未注明“稿件来源”的信息均来自网络整理。其文字、图片和音视频稿件的所属权归原作者所有。本站收集整理出于非商业性的教育和科研之目的,并不意味着本站赞同其观点或证实其内容的真实性。仅作为临时的测试数据,供内部测试之用。本站并未授权任何人以任何方式主动获取本站任何信息。

② 本站未注明“稿件来源”的临时测试数据将在测试完成后最终做删除处理。有问题或投稿请发送至: 邮箱/279061341@qq.com QQ/279061341

flink time and watermark

下载Word文档到电脑,方便收藏和打印~

下载Word文档

猜你喜欢

flink time and watermark

流处理中时间本质上就是一个普通的递增字段(long型,自1970年算起的微秒数),不一定真的表示时间。watermark只是应对乱序的办法之一,大多是启发式的,在延迟和完整性之间抉择。(如果没有延迟,就不够完整;如果有延迟,极端情况就是批处理,当然完整性足够高
2020-07-18

【赵强老师】Flink的Watermark机制(基于Flink 1.11.0实现)

在使用eventTime的时候如何处理乱序数据?我们知道,流处理从事件产生,到流经source,再到operator,中间是有一个过程和时间的。虽然大部分情况下,流到operator的数据都是按照事件产生的时间顺序来的,但是也不排除由于网络延迟等原因,导致乱序
【赵强老师】Flink的Watermark机制(基于Flink 1.11.0实现)
2022-03-21

Flink中的Time类型有哪些

这篇文章主要讲解了“Flink中的Time类型有哪些”,文中的讲解内容简单清晰,易于学习与理解,下面请大家跟着小编的思路慢慢深入,一起来研究和学习“Flink中的Time类型有哪些”吧!**Processing Time**Processi
2023-06-19

读MySQL 5.7文档11.2 Date and Time Data Types总结

作者:魏新平,知数堂第5期MySQL实战班学员,第10期MySQL优化班学员,现任职助教。读MySQL5.7文档11.2 Date and Time Data TypesMySQL的时间类型分为DATE, DATETIME, TIMESTAMP, TIME,
读MySQL 5.7文档11.2 Date and Time Data Types总结
2020-01-21

ORA-39359: Source time zone version is string and target time zone version is string. ORACLE 报错 故障修复

文档解释ORA-39359: Source time zone version is string and target time zone version is string.Cause: The time zone version
ORA-39359: Source time zone version is string and target time zone version is string. ORACLE 报错 故障修复
2023-11-05

ORA-39322: Cannot use transportable tablespace with TIMESTAMP WITH TIME ZONE columns and different t

文档解释ORA-39322: Cannot use transportable tablespace with TIMESTAMP WITH TIME ZONE columns and different time zone
ORA-39322: Cannot use transportable tablespace with TIMESTAMP WITH TIME ZONE columns and different t
2023-11-05

ORA-39361: Source time zone is string and target time zone is string. ORACLE 报错 故障修复 远程处理

文档解释ORA-39361: Source time zone is string and target time zone is string.Cause: The time zone of the source database is
ORA-39361: Source time zone is string and target time zone is string. ORACLE 报错 故障修复 远程处理
2023-11-05

ORA-39339: Table string was skipped due to transportable import and TSTZ issues resulting from time

文档解释ORA-39339: Table string was skipped due to transportable import and TSTZ issues resulting from time zone version
ORA-39339: Table string was skipped due to transportable import and TSTZ issues resulting from time
2023-11-05

ORA-39360: Table string was skipped due to transportable import and TSLTZ issues resulting from time

文档解释ORA-39360: Table string was skipped due to transportable import and TSLTZ issues resulting from time zone
ORA-39360: Table string was skipped due to transportable import and TSLTZ issues resulting from time
2023-11-05

ORA-01875: time zone minute must be between -59 and 59 ORACLE 报错 故障修复 远程处理

文档解释ORA-01875: time zone minute must be between -59 and 59Cause: The time zone minute specified was not in the valid
ORA-01875: time zone minute must be between -59 and 59 ORACLE 报错 故障修复 远程处理
2023-11-04

ORA-09319: slgtd: unable to obtain the current date and time ORACLE 报错 故障修复 远程处理

文档解释ORA-09319: slgtd: unable to obtain the current date and timeCause: The system time might be set incorrectlyAction:
ORA-09319: slgtd: unable to obtain the current date and time ORACLE 报错 故障修复 远程处理
2023-11-05

ORA-01874: time zone hour must be between -12 and 14 ORACLE 报错 故障修复 远程处理

文档解释ORA-01874: time zone hour must be between -12 and 14Cause: The time zone hour specified was not in the valid
ORA-01874: time zone hour must be between -12 and 14 ORACLE 报错 故障修复 远程处理
2023-11-04

编程热搜

目录