Flink SQL:Queries(Joins)
Joins
Batch Streaming
Flink SQL supports complex and flexible join operations over dynamic tables. There are several different types of joins to account for the wide variety of semantics queries may require.
Flink SQL支持动态表上复杂而灵活的join操作。有几种不同类型的join来解释可能需要的各种语义查询。
By default, the order of joins is not optimized. Tables are joined in the order in which they are specified in the FROM clause. You can tweak the performance of your join queries, by listing the tables with the lowest update frequency first and the tables with the highest update frequency last. Make sure to specify tables in an order that does not yield a cross join (Cartesian product), which are not supported and would cause a query to fail.
默认情况下，join顺序未优化。表按照FROM子句中指定的顺序join。您可以调整join查询的性能，方法是首先列出更新频率最低的表，然后列出更新频率最高的表。请确保以不产生交叉join(笛卡尔积)的顺序指定表，交叉join是不受支持的，会导致查询失败。
Regular Joins
Regular joins are the most generic type of join in which any new record, or changes to either side of the join, are visible and affect the entirety of the join result. For example, if there is a new record on the left side, it will be joined with all the previous and future records on the right side when the product id equals.
常规join是最通用的join类型，其中任何新记录或join任一侧的更改都是可见的，并且会影响整个join结果。例如，如果左侧有一条新记录，当product id相等时，它将与右侧所有先前和未来的记录join。
SELECT * FROM Orders
INNER JOIN Product
ON Orders.productId = Product.id
For streaming queries, the grammar of regular joins is the most flexible and allow for any kind of updating (insert, update, delete) input table. However, this operation has important operational implications: it requires to keep both sides of the join input in Flink state forever. Thus, the required state for computing the query result might grow infinitely depending on the number of distinct input rows of all input tables and intermediate join results. You can provide a query configuration with an appropriate state time-to-live (TTL) to prevent excessive state size. Note that this might affect the correctness of the query result. See query configuration for details.
对于流式查询，常规join的语法是最灵活的，允许任何类型的更新(insert, update, delete)输入表。然而，此操作具有重要的操作含义：它需要将join输入的两侧永远保持在Flink状态。因此，根据所有输入表和中间join结果的不同输入行数，计算查询结果所需的状态可能会无限增长。您可以为查询配置提供适当的状态生存时间(TTL)，以防止状态大小过大。请注意，这可能会影响查询结果的正确性。有关详细信息，请参阅查询配置。
For streaming queries the required state to compute the query result might grow infinitely depending on the type of aggregation and the number of distinct grouping keys. Please provide an idle state retention time to prevent excessive state size. See Idle State Retention Time for details.
对于流式查询，根据聚合类型和不同分组键的数量，计算查询结果所需的状态可能会无限增长。请提供空闲状态保留时间以防止状态大小过大。有关详细信息，请参阅空闲状态保留时间。
INNER Equi-JOIN
Returns a simple Cartesian product restricted by the join condition. Currently, only equi-joins are supported, i.e., joins that have at least one conjunctive condition with an equality predicate. Arbitrary cross or theta joins are not supported.
返回受join条件限制的简单笛卡尔积。目前，只支持等式连接，即至少有一个使用等式谓词的连接条件。不支持任意交叉或θ join。
SELECT *
FROM Orders
INNER JOIN Product
ON Orders.product_id = Product.id
OUTER Equi-JOIN
Returns all rows in the qualified Cartesian product (i.e., all combined rows that pass its join condition), plus one copy of each row in an outer table for which the join condition did not match with any row of the other table. Flink supports LEFT, RIGHT, and FULL outer joins. Currently, only equi-joins are supported, i.e., joins with at least one conjunctive condition with an equality predicate. Arbitrary cross or theta joins are not supported.
返回合格笛卡尔积中的所有行(即通过其join条件的所有联结行)，以及外部表中join条件与其他表的任何行不匹配的每一行的一个副本。Flink支持LEFT、RIGHT和FULL outer joins。目前，只支持等式联接，即至少有一个使用等式谓词的join条件。不支持任意交叉或θ连接。
SELECT *
FROM Orders
LEFT JOIN Product
ON Orders.product_id = Product.id
SELECT *
FROM Orders
RIGHT JOIN Product
ON Orders.product_id = Product.id
SELECT *
FROM Orders
FULL OUTER JOIN Product
ON Orders.product_id = Product.id
Interval Joins
Returns a simple Cartesian product restricted by the join condition and a time constraint. An interval join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Two appropriate range predicates can define such a condition (<, <=, >=, >), a BETWEEN predicate, or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.
返回受连接条件和时间约束限制的简单笛卡尔积。间隔连接至少需要一个相等连接谓词和一个限制双方时间的连接条件。两个适当的范围谓词可以这样定义: (<, <=, >=, >)、BETWEE谓词或一个比较两个输入表的相同类型(即处理时间或事件时间)的时间属性的相等谓词。
For example, this query will join all orders with their corresponding shipments if the order was shipped four hours after the order was received.
例如，如果在收到订单四小时后发货，此查询将连接所有订单及其相应的发货。
SELECT *
FROM Orders o, Shipments s
WHERE o.id = s.order_id
AND o.order_time BETWEEN s.ship_time - INTERVAL '4' HOUR AND s.ship_time
The following predicates are examples of valid interval join conditions:
以下谓词是有效的interval join条件的示例：
- ltime = rtime
- ltime >= rtime AND ltime < rtime + INTERVAL ‘10’ MINUTE
- ltime BETWEEN rtime - INTERVAL ‘10’ SECOND AND rtime + INTERVAL ‘5’ SECOND
For streaming queries, compared to the regular join, interval join only supports append-only tables with time attributes. Since time attributes are quasi-monotonic increasing, Flink can remove old values from its state without affecting the correctness of the result.
对于流查询，与常规联接相比，间隔联接仅支持具有时间属性的append-only表。由于时间属性是准单调递增的，Flink可以从其状态中删除旧值，而不会影响结果的正确性。
Temporal Joins
A Temporal table is a table that evolves over time - otherwise known in Flink as a dynamic table. Rows in a temporal table are associated with one or more temporal periods and all Flink tables are temporal(dynamic). The temporal table contains one or more versioned table snapshots, it can be a changing history table which tracks the changes(e.g. database changelog, contains all snapshots) or a changing dimensioned table which materializes the changes(e.g. database table which contains the latest snapshot).
临时表是一个随时间演变的表，在Flink中也称为动态表。临时表中的行与一个或多个临时周期相关联，并且所有Flink表都是临时的(动态的)。临时表包含一个或多个版本化的表快照，它可以是跟踪更改的更改历史表(例如数据库更改日志，包含所有快照)，也可以是实现更改的更改维度表(例如，包含最新快照的数据库表)。
Event Time Temporal Join
Event Time temporal joins allow joining against a versioned table. This means a table can be enriched with changing metadata and retrieve its value at a certain point in time.
事件时间临时联接允许针对版本化表进行联接。这意味着可以通过更改元数据来丰富表，并在某个时间点检索其值。
Temporal joins take an arbitrary table (left input/probe site) and correlate each row to the corresponding row’s relevant version in the versioned table (right input/build side). Flink uses the SQL syntax of FOR SYSTEM_TIME AS OF to perform this operation from the SQL:2011 standard. The syntax of a temporal join is as follows;
临时联接采用任意表(左输入/探测端)，并将每一行与版本化表(右输入/构建端)中相应行的相关版本相关联。Flink使用FOR SYSTEM_TIME AS OF 的SQL语法从SQL:2011标准执行此操作。临时连接的语法如下:
SELECT [column_list]
FROM table1 [AS <alias1>]
[LEFT] JOIN table2 FOR SYSTEM_TIME AS OF table1.{ proctime | rowtime } [AS <alias2>]
ON table1.column-name1 = table2.column-name1
With an event-time attribute (i.e., a rowtime attribute), it is possible to retrieve the value of a key as it was at some point in the past. This allows for joining the two tables at a common point in time. The versioned table will store all versions - identified by time - since the last watermark.
使用事件时间属性(即rowtime属性)，可以检索过去某个时间点的key值。这允许在同一时间点连接两个表。版本表将存储自上次水印以来的所有版本(按时间标识)。
For example, suppose we have a table of orders, each with prices in different currencies. To properly normalize this table to a single currency, such as USD, each order needs to be joined with the proper currency conversion rate from the point-in-time when the order was placed.
例如，假设我们有一个订单表，每个订单都有不同货币的价格。为了将此表正确规范化为单一货币，例如美元，每个订单都需要从下订单的时间点起加入适当的货币兑换率。
-- Create a table of orders. This is a standard
-- append-only dynamic table.
CREATE TABLE orders (
order_id STRING,
price DECIMAL(32,2),
currency STRING,
order_time TIMESTAMP(3),
WATERMARK FOR order_time AS order_time
) WITH (/* ... */);
-- Define a versioned table of currency rates.
-- This could be from a change-data-capture
-- such as Debezium, a compacted Kafka topic, or any other
-- way of defining a versioned table.
CREATE TABLE currency_rates (
currency STRING,
conversion_rate DECIMAL(32, 2),
update_time TIMESTAMP(3) METADATA FROM `values.source.timestamp` VIRTUAL,
WATERMARK FOR update_time AS update_time,
PRIMARY KEY(currency) NOT ENFORCED
) WITH (
'connector' = 'kafka',
'value.format' = 'debezium-json',
/* ... */
);
SELECT
order_id,
price,
currency,
conversion_rate,
order_time
FROM orders
LEFT JOIN currency_rates FOR SYSTEM_TIME AS OF orders.order_time
ON orders.currency = currency_rates.currency;
order_id price currency conversion_rate order_time
======== ===== ======== =============== =========
o_001 11.11 EUR 1.14 12:00:00
o_002 12.51 EUR 1.10 12:06:00
Note: The event-time temporal join is triggered by a watermark from the left and right sides; please ensure both sides of the join have set watermark correctly.
注意：事件时间临时连接由左右两侧的水印触发；请确保连接的两侧都正确设置了水印。
Note: The event-time temporal join requires the primary key contained in the equivalence condition of the temporal join condition, e.g., The primary key currency_rates.currency of table currency_rates to be constrained in the condition orders.currency = currency_rates.currency.
注意：事件时间临时连接需要临时连接条件的相等条件中包含主键，例如表currencyrates的主键currency_rates.currency 在条件orders.currency = currency_rates.currency中被约束。
In contrast to regular joins, the previous temporal table results will not be affected despite the changes on the build side. Compared to interval joins, temporal table joins do not define a time window within which the records will be joined. Records from the probe side are always joined with the build side’s version at the time specified by the time attribute. Thus, rows on the build side might be arbitrarily old. As time passes, no longer needed versions of the record (for the given primary key) will be removed from the state.
与常规联接不同，尽管在构建端进行了更改，但之前的临时表结果不会受到影响。与interval joins相比，临时表联接没有定义记录联接的时间窗口。探测端的记录总是在time属性指定的时间与构建端的版本连接。因此，构建端的行可能是任意旧的。随着时间的推移，不再需要的记录版本(对于给定的主键)将从状态中删除。
Processing Time Temporal Join
A processing time temporal table join uses a processing-time attribute to correlate rows to the latest version of a key in an external versioned table.
处理时间临时表联接使用处理时间属性将行与外部版本表中key的最新版本关联起来。
By definition, with a processing-time attribute, the join will always return the most up-to-date value for a given key. One can think of a lookup table as a simple HashMap
根据定义，使用处理时间属性，联接将始终返回给定key的最新值。可以将lookup表看作一个简单的HashMap
The following processing-time temporal table join example shows an append-only table orders that should be joined with the table LatestRates. LatestRates is a dimension table (e.g. HBase table) that is materialized with the latest rate. At time 10:15, 10:30, 10:52, the content of LatestRates looks as follows:
下面的处理时间临时表连接示例显示了一个append-only表orders，该订单应与表LatestRates连接。LatestRates是一个维度表(例如HBase表)，用最新的速率具体化。在时间10:15、10:30和10:52，LatestRates的内容如下：
10:15> SELECT * FROM LatestRates;
currency rate
======== ======
US Dollar 102
Euro 114
Yen 1
10:30> SELECT * FROM LatestRates;
currency rate
======== ======
US Dollar 102
Euro 114
Yen 1
10:52> SELECT * FROM LatestRates;
currency rate
======== ======
US Dollar 102
Euro 116 <==== changed from 114 to 116
Yen 1
The content of LastestRates at times 10:15 and 10:30 are equal. The Euro rate has changed from 114 to 116 at 10:52.
时间10:15和10:30的LastRates的内容相等。欧元汇率在10:52从114变为116。
Orders is an append-only table representing payments for the given amount and the given currency. For example, at 10:15 there was an order for an amount of 2 Euro.
订单是一个append-only表，表示给定金额和给定货币的付款。例如，在10点15分，有一笔金额为2欧元的订单。
SELECT * FROM Orders;
amount currency
====== =========
2 Euro <== arrived at time 10:15
1 US Dollar <== arrived at time 10:30
2 Euro <== arrived at time 10:52
Given these tables, we would like to calculate all Orders converted to a common currency.
根据这些表格，我们希望计算所有转换为通用货币的订单。
amount currency rate amount*rate
====== ========= ======= ============
2 Euro 114 228 <== arrived at time 10:15
1 US Dollar 102 102 <== arrived at time 10:30
2 Euro 116 232 <== arrived at time 10:52
Currently, the FOR SYSTEM_TIME AS OF syntax used in temporal join with latest version of any view/table is not support yet, you can use temporal table function syntax as following:
目前，在与任何视图/表的最新版本的临时连接中使用的FOR SYSTEM_TIME AS OF语法尚不受支持，您可以按如下方式使用临时表函数语法：
SELECT
o_amount, r_rate
FROM
Orders,
LATERAL TABLE (Rates(o_proctime))
WHERE
r_currency = o_currency
Note The reason why the FOR SYSTEM_TIME AS OF syntax used in temporal join with latest version of any table/view is not support is only the semantic consideration, because the join processing for left stream doesn’t wait for the complete snapshot of temporal table, this may mislead users in production environment. The processing-time temporal join by temporal table function also exists same semantic problem, but it has been alive for a long time, thus we support it from the perspective of compatibility.
注意：对于任何表/视图的最新版本，临时连接中使用的FOR SYSTEM_TIME AS OF语法不受支持的原因仅是语义考虑，因为左流的连接处理不等待临时表的完整快照，这可能会误导生产环境中的用户。临时表函数的处理时间临时连接也存在同样的语义问题，但它已经存在了很长时间，因此我们从兼容性的角度来支持它。
The result is not deterministic for processing-time. The processing-time temporal join is most often used to enrich the stream with an external table (i.e., dimension table).
结果对于处理时间来说是不确定的。处理时间临时连接最常用于使用外部表(即维度表)来丰富流。
In contrast to regular joins, the previous temporal table results will not be affected despite the changes on the build side. Compared to interval joins, temporal table joins do not define a time window within which the records join, i.e., old rows are not stored in state.
与常规联接不同，尽管在构建端进行了更改，但之前的临时表结果不会受到影响。与interval joins相比，临时表联接没有定义记录联接的时间窗口，即，旧行不会以状态存储。
Temporal Table Function Join
The syntax to join a table with a temporal table function is the same as in Join with Table Function.
使用临时表函数连接表的语法与使用表函数连接相同。
Note: Currently only inner join and left outer join with temporal tables are supported.
注意：目前只支持临时表的inner join and left outer join。
Assuming Rates is a temporal table function, the join can be expressed in SQL as follows:
假设Rates是一个临时表函数，连接可以用SQL表示如下：
SELECT
o_amount, r_rate
FROM
Orders,
LATERAL TABLE (Rates(o_proctime))
WHERE
r_currency = o_currency
The main difference between above Temporal Table DDL and Temporal Table Function are:
上述临时表DDL和临时表函数的主要区别是：
- The temporal table DDL can be defined in SQL but temporal table function can not;
临时表DDL可以在SQL中定义，但临时表函数不能； - Both temporal table DDL and temporal table function support temporal join versioned table, but only temporal table function can temporal join the latest version of any table/view.
临时表DDL和临时表函数都支持临时连接版本表，但只有临时表函数才能临时连接任何表/视图的最新版本。
Lookup Join
A lookup join is typically used to enrich a table with data that is queried from an external system. The join requires one table to have a processing time attribute and the other table to be backed by a lookup source connector.
lookup join通常用于使用从外部系统查询的数据来丰富表。联接要求一个表具有处理时间属性，另一个表由a lookup source connector支持。
The lookup join uses the above Processing Time Temporal Join syntax with the right table to be backed by a lookup source connector.
lookup join使用上面的“Processing Time Temporal Join”语法，右表由a lookup source connector支持。
The following example shows the syntax to specify a lookup join.
以下示例显示了指定lookup join的语法。
-- Customers is backed by the JDBC connector and can be used for lookup joins
CREATE TEMPORARY TABLE Customers (
id INT,
name STRING,
country STRING,
zip STRING
) WITH (
'connector' = 'jdbc',
'url' = 'jdbc:mysql://mysqlhost:3306/customerdb',
'table-name' = 'customers'
);
-- enrich each order with customer information
SELECT o.order_id, o.total, c.country, c.zip
FROM Orders AS o
JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
ON o.customer_id = c.id;
In the example above, the Orders table is enriched with data from the Customers table which resides in a MySQL database. The FOR SYSTEM_TIME AS OF clause with the subsequent processing time attribute ensures that each row of the Orders table is joined with those Customers rows that match the join predicate at the point in time when the Orders row is processed by the join operator. It also prevents that the join result is updated when a joined Customer row is updated in the future. The lookup join also requires a mandatory equality join predicate, in the example above o.customer_id = c.id.
在上面的示例中，Orders表丰富了来自MySQL数据库中Customers表的数据。带有处理时间属性的FOR SYSTEM_TIME AS OF子句可确保Orders表的每一行与那些在join算子处理Orders行时与连接谓词匹配的Customers表的行连接。它还防止在将来更新joined Customer行时更新join结果。lookup join还需要一个强制的相等联接谓词，在上面的示例中，o.customer_id＝c.id。
Array Expansion
Returns a new row for each element in the given array. Unnesting WITH ORDINALITY is not yet supported.
返回给定数组中每个元素的新行。尚不支持Unnesting WITH ORDINALITY。
SELECT order_id, tag
FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
Table Function
Joins a table with the results of a table function. Each row of the left (outer) table is joined with all rows produced by the corresponding call of the table function. User-defined table functions must be registered before use.
将表与表函数的结果联接。left (outer)表的每一行都与表函数的相应调用产生的所有行连接。用户定义的表函数必须在使用前注册。
INNER JOIN
The row of the left (outer) table is dropped, if its table function call returns an empty result.
如果left (outer) 表的表函数调用返回空结果，则该行将被删除。
SELECT order_id, res
FROM Orders,
LATERAL TABLE(table_func(order_id)) t(res)
LEFT OUTER JOIN
If a table function call returns an empty result, the corresponding outer row is preserved, and the result padded with null values. Currently, a left outer join against a lateral table requires a TRUE literal in the ON clause.
如果表函数调用返回空结果，则保留相应的外部行，并用空值填充结果。目前，针对a lateral table的a left outer join需要ON子句中的TRUE文本。
SELECT order_id, res
FROM Orders
LEFT OUTER JOIN LATERAL TABLE(table_func(order_id)) t(res)
ON TRUE