Comprehensions and Generators（生成器）
- Item27: 用Comprehensions而不是map和filter
当我们想要计算数组的元素平方时
a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
squares = []
for x in a:
squares.append(x**2)
print(squares)
>>>
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
可以用Comprehension实现一样的结果
squares = [x**2 for x in a] # List comprehension
print(squares)
>>>
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
除非应用单参数的函数，列表comprehension比map更加清晰。
map需要构建lambda函数，视觉上复杂。
alt = map(lambda x: x ** 2, a)
当然也可以增加限定条件
even_squares = [x**2 for x in a if x % 2 == 0]
print(even_squares)
>>>
[4, 16, 36, 64, 100]
map配合filter也是可以的。
alt = map(lambda x: x**2, filter(lambda x: x % 2 == 0, a))
assert even_squares == list(alt)
字典和集合也是可以使用comprehension的。
even_squares_dict = {x: x**2 for x in a if x % 2 == 0}
threes_cubed_set = {x**3 for x in a if x % 3 == 0}
print(even_squares_dict)
print(threes_cubed_set)
>>>
{2: 4, 4: 16, 6: 36, 8: 64, 10: 100}
{216, 729, 27}
用dict和set来配合map和filter进行构建也是可以的，但是语句偏长，需要换行，显得不优雅。
alt_dict = dict(map(lambda x: (x, x**2),
filter(lambda x: x % 2 == 0, a)))
alt_set = set(map(lambda x: x**3,
filter(lambda x: x % 3 == 0, a)))
- Item28: 避免在Comprehensions里面使用超过两个的控制表达式
Comprehension支持多层的循环。例如一个简单的矩阵，转换成列表。
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat = [x for row in matrix for x in row]
print(flat)
>>>
[1, 2, 3, 4, 5, 6, 7, 8, 9]
只要添加额外的[]字符，也可以构建出对应的二维矩阵：
squared = [[x**2 for x in row] for row in matrix]
print(squared)
>>>
[[1, 4, 9], [16, 25, 36], [49, 64, 81]]
如果有三层或更多层的嵌套，则不适合用Comprehension了。
my_lists = [
[[1, 2, 3], [4, 5, 6]],
...
]
flat = [x for sublist1 in my_lists
for sublist2 in sublist1
for x in sublist2]
此时，使用普通的for-loop反倒更清晰：
flat = []
for sublist1 in my_lists:
for sublist2 in sublist1:
flat.extend(sublist2)
Comprehension可以有多个判断条件
a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b = [x for x in a if x > 4 if x % 2 == 0]
c = [x for x in a if x > 4 and x % 2 == 0]
超过两个控制语句就尽量不要用Comprehension了。以下反例就是：找到行相加不小于10且行中整除3的元素。
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
filtered = [[x for x in row if x % 3 == 0]
for row in matrix if sum(row) >= 10]
print(filtered)
>>>
[[6], [9]]
- Item29: 通过使用赋值表达式，避免Comprehensions里的重复工作
以下是一个简单的例子：零件工厂供货。(8个零件为一批，最后得到各个零件的批次数)
stock = {
'nails': 125,
'screws': 35,
'wingnuts': 8,
'washers': 24,
}
order = ['screws', 'wingnuts', 'clips']
def get_batches(count, size):
return count // size
result = {}
for name in order:
count = stock.get(name, 0)
batches = get_batches(count, 8)
if batches:
result[name] = batches
print(result)
>>>
{'screws': 4, 'wingnuts': 1}
可以通过Compresion的方式来紧凑这个表达，但是存在重复的计算。
found = {name: get_batches(stock.get(name, 0), 8)
for name in order
if get_batches(stock.get(name, 0), 8)}
print(found)
>>>
{'screws': 4, 'wingnuts': 1}
如果其中一个调用出错，则整体出错。（前后一致性差）
has_bug = {name: get_batches(stock.get(name, 0), 4)
for name in order
if get_batches(stock.get(name, 0), 8)}
print('Expected:', found)
print('Found: ', has_bug)
>>>
Expected: {'screws': 4, 'wingnuts': 1}
Found: {'screws': 8, 'wingnuts': 2}
Python3.8之后的海象赋值操作可以缓解：
found = {name: batches for name in order
if (batches := get_batches(stock.get(name, 0), 8))}
:=的顺序要用对，不然会报错：
result = {name: (tenth := count // 10)
for name, count in stock.items() if tenth > 0}
>>>
Traceback ...
NameError: name 'tenth' is not defined
result = {name: tenth for name, count in stock.items()
if (tenth := count // 10) > 0}
print(result)
>>>
{'nails': 12, 'screws': 3, 'washers': 2}
:=可能会泄露变量到外面的作用域。
half = [(last := count // 2) for count in
stock.values()]
print(f'Last item of {half} is {last}')
>>>
Last item of [62, 17, 4, 12] is 12
当然，for-loop也会
for count in stock.values(): # Leaks loop variable
pass
print(f'Last item of {list(stock.values())} is {count}')
>>>
Last item of [125, 35, 8, 24] is 24
Comprehension就不会。
half = [count // 2 for count in stock.values()]
print(half) # Works
print(count) # Exception because loop variable didn't leak
>>>
[62, 17, 4, 12]
Traceback ...
NameError: name 'count' is not defined
found = ((name, batches) for name in order
if (batches := get_batches(stock.get(name, 0), 8)))
print(next(found))
print(next(found))
>>>
('screws', 4)
('wingnuts', 1)
- Item30: 考虑使用生成器（Generators）而不是返回列表
找到下一个单词的下标：
def index_words(text):
result = []
if text:
result.append(0)
for index, letter in enumerate(text):
if letter == ' ':
result.append(index + 1)
return result
address = 'Four score and seven years ago...'
result = index_words(address)
print(result[:10])
>>>
[0, 5, 11, 15, 21, 27, 31, 35, 43, 51]
存在两个问题：1、代码稍显冗余
可以用generator来代替：
def index_words_iter(text):
if text:
yield 0
for index, letter in enumerate(text):
if letter == ' ':
yield index + 1
next可以取到下一个数值：
it = index_words_iter(address)
print(next(it))
print(next(it))
>>>
0
5
或者用list取到所有数值：
result = list(index_words_iter(address))
print(result[:10])
>>>
[0, 5, 11, 15, 21, 27, 31, 35, 43, 51]
2、可能由于过大的list导致内存不足，而generator一次一行地yield，则好一些：
def index_file(handle):
offset = 0
for line in handle:
if line:
yield offset
for letter in line:
offset += 1
if letter == ' ':
yield offset
读取文件，利用itertools.islice获得最后的10个结果。
with open('address.txt', 'r') as f:
it = index_file(f)
results = itertools.islice(it, 0, 10)
print(list(results))
>>>
[0, 5, 11, 15, 21, 27, 31, 35, 43, 51]
- Item31: 当迭代作为参数的时候，要防备bug发生
与list有关的应用，通常都需要多次迭代列表，比如统计数字的占比：
def normalize(numbers):
total = sum(numbers)
result = []
for value in numbers:
percent = 100 * value / total
result.append(percent)
return result
visits = [15, 35, 80]
percentages = normalize(visits)
print(percentages)
assert sum(percentages) == 100.0
>>>
[11.538461538461538, 26.923076923076923, 61.53846153846154]
比如每次一行yield这些数字，然后交给函数normalize，结果发现是空list：
def read_visits(data_path):
with open(data_path) as f:
for line in f:
yield int(line)
it = read_visits('my_numbers.txt')
percentages = normalize(it)
print(percentages)
>>>
[]
因为sum的时候已经消耗了一次，再次迭代的时候，已经是空列表了：
it = read_visits('my_numbers.txt')
print(list(it))
print(list(it)) # Already exhausted
>>>
[15, 35, 80]
[]
解决方案是：自己先手动备份一次list下来，然后用这个list，确保迭代不会遇到这个问题：
def normalize_copy(numbers):
numbers_copy = list(numbers) # Copy the iterator
total = sum(numbers_copy)
result = []
for value in numbers_copy:
percent = 100 * value / total
result.append(percent)
return result
it = read_visits('my_numbers.txt')
percentages = normalize_copy(it)
print(percentages)
assert sum(percentages) == 100.0
>>>
[11.538461538461538, 26.923076923076923, 61.53846153846154]
但是这个解决方案，可能会导致生成的list过大，导致内存不足。另一种解决方案是每次都生成一个新的迭代器：
def normalize_func(get_iter):
total = sum(get_iter()) # New iterator
result = []
for value in get_iter(): # New iterator
percent = 100 * value / total
result.append(percent)
return result
path = 'my_numbers.txt'
percentages = normalize_func(lambda: read_visits(path))
print(percentages)
assert sum(percentages) == 100.0
>>>
[11.538461538461538, 26.923076923076923, 61.53846153846154]
当Python看到比如for x in foo的语句,它实际上调用iter(foo).这个iter又调用了foo.__iter__。这个__iter__ 方法应该返回一个迭代器对象(实现了__next__ 方法)。然后循环地调用这个方法直到(抛出StopIteration异常)。
听着很复杂，实现一个：
class ReadVisits:
def __init__(self, data_path):
self.data_path = data_path
def __iter__(self):
with open(self.data_path) as f:
for line in f:
yield int(line)
这样可以确保每次调用ReadVisits.__iter__都会得到一个新的迭代器
visits = ReadVisits(path)
percentages = normalize(visits)
print(percentages)
assert sum(percentages) == 100.0
>>>
[11.538461538461538, 26.923076923076923, 61.53846153846154]
通过iter获得迭代器，如果是本身的话，说明传递了一个迭代器进来，直接抛出TypeError：
def normalize_defensive(numbers):
if iter(numbers) is numbers: # An iterator -- bad!
raise TypeError('Must supply a container')
total = sum(numbers)
result = []
for value in numbers:
percent = 100 * value / total
result.append(percent)
return result
也可以通过collections.abc来判断是不是迭代器：
from collections.abc import Iterator
def normalize_defensive(numbers):
if isinstance(numbers, Iterator): # Another way to check
raise TypeError('Must supply a container')
total = sum(numbers)
result = []
for value in numbers:
percent = 100 * value / total
result.append(percent)
return result
visits = [15, 35, 80]
percentages = normalize_defensive(visits)
assert sum(percentages) == 100.0
visits = ReadVisits(path)
percentages = normalize_defensive(visits)
assert sum(percentages) == 100.0
visits = [15, 35, 80]
it = iter(visits)
normalize_defensive(it)
>>>
Traceback ...
TypeError: Must supply a container
- Item32: 使用大的列表Comprohensions的时候，考虑采用生成器表达式
为了解决这个问题，比如处理文件时：
value = [len(x) for x in open('my_file.txt')]
print(value)
>>>
[100, 57, 15, 1, 12, 75, 5, 86, 89, 11]
可以使用generator表达式，通过next来访问：
it = (len(x) for x in open('my_file.txt'))
print(it)
>>>
at 0x108993dd0>
print(next(it))
print(next(it))
>>>
100
57
另一个强力的输出是，generator可以组合使用：
roots = ((x, x**0.5) for x in it)
print(next(roots))
>>>
(15, 3.872983346207417)
- Item33: 用yield from来组合多个生成器
比如需要生成动画：
def move(period, speed):
for _ in range(period):
yield speed
def pause(delay):
for _ in range(delay):
yield 0
def animate():
for delta in move(4, 5.0):
yield delta
for delta in pause(3):
yield delta
for delta in move(2, 3.0):
yield delta
其中，以5的速度动4秒，停3秒，以3的速度动2秒。然后下面进行渲染：
def render(delta):
print(f'Delta: {delta:.1f}')
# Move the images onscreen
...
def run(func):
for delta in func():
render(delta)
run(animate)
>>>
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 0.0
Delta: 0.0
Delta: 0.0
Delta: 3.0
Delta: 3.0
但是，animate显得冗余，可以用下面的yield from来组合动画：
def animate_composed():
yield from move(4, 5.0)
yield from pause(3)
yield from move(2, 3.0)
run(animate_composed)
>>>
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 0.0
Delta: 0.0
Delta: 0.0
Delta: 3.0
Delta: 3.0
import timeit
def child():
for i in range(1_000_000):
yield i
def slow():
for i in child():
yield i
def fast():
yield from child()
baseline = timeit.timeit(
stmt='for _ in slow(): pass',
globals=globals(),
number=50)
print(f'Manual nesting {baseline:.2f}s')
comparison = timeit.timeit(
stmt='for _ in fast(): pass',
globals=globals(),
number=50)
print(f'Composed nesting {comparison:.2f}s')
reduction = -(comparison - baseline) / baseline
print(f'{reduction:.1%} less time')
>>>
Manual nesting 4.02s
Composed nesting 3.47s
13.5% less time
yield from当成for-yield来用，性能有提升。
- Item34: 避免用send插入数据到生成器
加入现在有一个生成sin波的函数：
import math
def wave(amplitude, steps):
step_size = 2 * math.pi / steps
for step in range(steps):
radians = step * step_size
fraction = math.sin(radians)
output = amplitude * fraction
yield output
还有发射函数：
def transmit(output):
if output is None:
print(f'Output is None')
else:
print(f'Output: {output:>5.1f}')
def run(it):
for output in it:
transmit(output)
run(wave(3.0, 8))
>>>
Output: 0.0
Output: 2.1
Output: 3.0
Output: 2.1
Output: 0.0
Output: -2.1
Output: -3.0
Output: -2.1
产生正常的sin波可以，但是如果想在中间调整振幅来达到AM波就不行。
def my_generator():
received = yield 1
print(f'received = {received}')
it = iter(my_generator())
output = next(it) # Get first generator output
print(f'output = {output}')
try:
next(it) # Run generator until it exits
except StopIteration:
pass
>>>
output = 1
received = None
send方法提供了一种双向通信的方式
it = iter(my_generator())
output = it.send(None) # Get first generator output
print(f'output = {output}')
try:
it.send('hello!') # Send value into the generator
except StopIteration:
pass
>>>
output = 1
received = hello!
通过在yield之后接收一个数值：
def wave_modulating(steps):
step_size = 2 * math.pi / steps
amplitude = yield # Receive initial amplitude
for step in range(steps):
radians = step * step_size
fraction = math.sin(radians)
output = amplitude * fraction
amplitude = yield output # Receive next amplitude
def run_modulating(it):
amplitudes = [
None, 7, 7, 7, 2, 2, 2, 2, 10, 10, 10, 10, 10]
for amplitude in amplitudes:
output = it.send(amplitude)
transmit(output)
run_modulating(wave_modulating(12))
>>>
Output is None
Output: 0.0
Output: 3.5
Output: 6.1
Output: 2.0
Output: 1.7
Output: 1.0
Output: 0.0
Output: -5.0
Output: -8.7
Output: -10.0
Output: -8.7
Output: -5.0
当有多个波的时候，用yield from组合是部分可行的。
def complex_wave():
yield from wave(7.0, 3)
yield from wave(2.0, 4)
yield from wave(10.0, 5)
run(complex_wave())
>>>
Output: 0.0
Output: 6.1
Output: -6.1
Output: 0.0
Output: 2.0
Output: 0.0
Output: -2.0
Output: 0.0
Output: 9.5
Output: 5.9
Output: -5.9
Output: -9.5
但是如果yield from和send组合在一块，就会显得十分复杂，不适合阅读。而且需要修改大量的代码。
def complex_wave_modulating():
yield from wave_modulating(3)
yield from wave_modulating(4)
yield from wave_modulating(5)
run_modulating(complex_wave_modulating())
>>>
Output is None
Output: 0.0
Output: 6.1
Output: -6.1
Output is None
Output: 0.0
Output: 2.0
Output: 0.0
Output: -10.0
Output is None
Output: 0.0
Output: 9.5
Output: 5.9
解决方案是：从wave生成的部分进行修改：
def wave_cascading(amplitude_it, steps):
step_size = 2 * math.pi / steps
for step in range(steps):
radians = step * step_size
fraction = math.sin(radians)
amplitude = next(amplitude_it) # Get next input
output = amplitude * fraction
yield output
再和yield from进行组合，就不会有问题：
def complex_wave_cascading(amplitude_it):
yield from wave_cascading(amplitude_it, 3)
yield from wave_cascading(amplitude_it, 4)
yield from wave_cascading(amplitude_it, 5)
def run_cascading():
amplitudes = [7, 7, 7, 2, 2, 2, 2, 10, 10, 10, 10, 10]
it = complex_wave_cascading(iter(amplitudes))
for amplitude in amplitudes:
output = next(it)
transmit(output)
run_cascading()
>>>
Output: 0.0
Output: 6.1
Output: -6.1
Output: 0.0
Output: 2.0
Output: 0.0
Output: -2.0
Output: 0.0
Output: 9.5
Output: 5.9
Output: -5.9
Output: -9.5
唯一的缺点是此代码的输入生成器假定是完全线程安全的，但情况可能并非如此。如果需要跨越线程边界，异步函数可能更合适。
- Item35: 在生成器中，避免使用throw来造成状态转换（State Transitions）
当调用throw方法时，下一次出现的 yield 表达式会在接收到其输出后重新引发提供的 Exception 实例，而不是正常继续。看简单实例：
class MyError(Exception):
pass
def my_generator():
yield 1
yield 2
yield 3
it = my_generator()
print(next(it)) # Yield 1
print(next(it)) # Yield 2
print(it.throw(MyError('test error')))
>>>
1
2
Traceback ...
MyError: test error
当你调用 throw 时，生成器函数可能会使用标准的 try/except 复合语句捕获注入的异常，该复合语句围绕最后执行的 yield 表达式。
def my_generator():
yield 1
try:
yield 2
except MyError:
print('Got MyError!')
else:
yield 3
yield 4
it = my_generator()
print(next(it)) # Yield 1
print(next(it)) # Yield 2
print(it.throw(MyError('test error')))
>>>
1
2
Got MyError!
4
可以利用throw这一点来控制计时器的reset
class Reset(Exception):
pass
def timer(period):
current = period
while current:
current -= 1
try:
yield current
except Reset:
current = period
def check_for_reset():
# Poll for external event
...
def announce(remaining):
print(f'{remaining} ticks remaining')
def run():
it = timer(4)
while True:
try:
if check_for_reset():
current = it.throw(Reset())
else:
current = next(it)
except StopIteration:
break
else:
announce(current)
run()
>>>
3 ticks remaining
2 ticks remaining
1 ticks remaining
3 ticks remaining
2 ticks remaining
3 ticks remaining
2 ticks remaining
1 ticks remaining
0 ticks remaining
上面的代码还是稍显复杂，一个更简单的方式是实现状态闭包：
class Timer:
def __init__(self, period):
self.current = period
self.period = period
def reset(self):
self.current = self.period
def __iter__(self):
while self.current:
self.current -= 1
yield self.current
代码更简单易懂，尽可能不用throw方法。
def run():
timer = Timer(4)
for current in timer:
if check_for_reset():
timer.reset()
announce(current)
run()
>>>
3 ticks remaining
2 ticks remaining
1 ticks remaining
3 ticks remaining
2 ticks remaining
3 ticks remaining
2 ticks remaining
1 ticks remaining
0 ticks remaining
- Item36: 考虑用itertools来配合迭代器和生成器工作
导入包
import itertools
主要这个库包含三大点：
- 连接迭代器们
chain：连接多个迭代器：
it = itertools.chain([1, 2, 3], [4, 5, 6])
print(list(it))
>>>
[1, 2, 3, 4, 5, 6]
用repeat来输出单值：
it = itertools.repeat('hello', 3)
print(list(it))
>>>
['hello', 'hello', 'hello']
用cycle来重复迭代项目：
it = itertools.cycle([1, 2])
result = [next(it) for _ in range (10)]
print(result)
>>>
[1, 2, 1, 2, 1, 2, 1, 2, 1, 2]
用tee来拆分单个迭代器到多个并行迭代器。（如果迭代器没有以相同的速度前进，则此函数的内存使用量将增加，因为将需要缓冲来排队待处理的项目。）
it1, it2, it3 = itertools.tee(['first', 'second'], 3)
print(list(it1))
print(list(it2))
print(list(it3))
>>>
['first', 'second']
['first', 'second']
['first', 'second']
zip_longest来处理不等长的zip
keys = ['one', 'two', 'three']
values = [1, 2]
normal = list(zip(keys, values))
print('zip: ', normal)
it = itertools.zip_longest(keys, values, fillvalue='nope')
longest = list(it)
print('zip_longest:', longest)
>>>
zip: [('one', 1), ('two', 2)]
zip_longest: [('one', 1), ('two', 2), ('three', 'nope')]
- 过滤项目
islice：指定end或者start和end或者start,end,step_sizes来切分（不复制）。
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
first_five = itertools.islice(values, 5)
print('First five: ', list(first_five))
middle_odds = itertools.islice(values, 2, 8, 2)
print('Middle odds:', list(middle_odds))
>>>
First five: [1, 2, 3, 4, 5]
Middle odds: [3, 5, 7]
takewhile，直到函数返回False才停止
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
less_than_seven = lambda x: x < 7
it = itertools.takewhile(less_than_seven, values)
print(list(it))
>>>
[1, 2, 3, 4, 5, 6]
dropwhile：是takewhile的相反，跳过直到返回True
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
less_than_seven = lambda x: x < 7
it = itertools.dropwhile(less_than_seven, values)
print(list(it))
>>>
[7, 8, 9, 10]
filterfalse：filter的相反，返回所有为False的项目
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
evens = lambda x: x % 2 == 0
filter_result = filter(evens, values)
print('Filter: ', list(filter_result))
filter_false_result = itertools.filterfalse(evens, values)
print('Filter false:', list(filter_false_result))
>>>
Filter: [2, 4, 6, 8, 10]
Filter false: [1, 3, 5, 7, 9]
- 产生组合
accumulate：和reduce功能相似，如果不指定参数，就是sum。
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
sum_reduce = itertools.accumulate(values)
print('Sum: ', list(sum_reduce))
def sum_modulo_20(first, second):
output = first + second
return output % 20
modulo_reduce = itertools.accumulate(values, sum_modulo_20)
print('Modulo:', list(modulo_reduce))
>>>
Sum: [1, 3, 6, 10, 15, 21, 28, 36, 45, 55]
Modulo: [1, 3, 6, 10, 15, 1, 8, 16, 5, 15]
product：产生笛卡尔积
single = itertools.product([1, 2], repeat=2)
print('Single: ', list(single))
multiple = itertools.product([1, 2], ['a', 'b'])
print('Multiple:', list(multiple))
>>>
Single: [(1, 1), (1, 2), (2, 1), (2, 2)]
Multiple: [(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')]
permutations：产生排列
it = itertools.permutations([1, 2, 3, 4], 2)
print(list(it))
>>>
[(1, 2),
(1, 3),
(1, 4),
(2, 1),
(2, 3),
(2, 4),
(3, 1),
(3, 2),
(3, 4),
(4, 1),
(4, 2),
(4, 3)]
combinations：产生组合
it = itertools.combinations([1, 2, 3, 4], 2)
print(list(it))
>>>
[(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)]
combinations_with_replacement：产生有放回的组合。（单个元素无限次取出）
it = itertools.combinations_with_replacement([1, 2, 3, 4], 2)
print(list(it))
>>>
[(1, 1),
(1, 2),
(1, 3),
(1, 4),
(2, 2),
(2, 3),
(2, 4),
(3, 3),
(3, 4),
(4, 4)]